Changing the configure script should not be necessary. The xotcl configure scripts conform exactly to the TEA standards. Removing TCL_DBGX hints that tcl itself has an installation problem concerning the debug versions of the tcl libraries.
David is right that the two current xotcl-core and xowiki versions (in the oacs-5-3 and in head) are not compatible with oacs 5.2.* due to changes between oacs-5-2 and oacs-5-3. I don't have the time working on this in the near future. If you have big interest, there are companies willing and capable to do this for e.g. the version in the 5.3 branch with little effort (send me email if you are interested).
It is possible to install old versions of xotcl-core and xowiki on 5.2, either from cvs or - as nima points out - from the download area on media. The apm files in the download area of media above are very old. If you have cvs access, i have heard that using the version form march works well with oacs-5-2
cvs -z3 -d:pserver:mailto:anonymous@cvs.openacs.org:/cvsroot co -D 2007-03-03 openacs-4/packages/xotcl-core
cvs -z3 -d:pserver:mailto:anonymous@cvs.openacs.org:/cvsroot co -D 2007-03-03 openacs-4/packages/xowiki
hope that helps
