I had a similar problem with my setup. I guess you're trying to load the tDOM module from the config.tcl file with a line like:
ns_param libtdom /path/to/libtdom0.7.8.so
The problem you have is that tDOM isn't compiled for AOLServer, so the module initialisation procedure isn't present. But on AOLServer 4 you don't need to compile tDOM with AOLServer support, you can load it directly using the 'package require' mechanism. I did this by taking the line above out of config.tcl and creating a file called tdom.tcl in my aolserver modules directory (/usr/lib/aolserver/modules/tcl on my system, /usr/local/aolserver/modules/tcl on yours I guess) with the following line in:
ns_eval package require tdom
That should cause AOLServer to load the tDOM module on startup, so its ready for use by OpenACS. Maybe this should be in the install docs somewhere? Hope that helps.