Forum OpenACS Q&A: Outdated XML libraries?

Collapse
1: Outdated XML libraries?
Posted by Bart Teeuwisse on 03/06/03 12:07 AM
I've started replacing ns_xml in favor of tDOM and discovered what appears to be legacy code from before ns_xml.

Originally tcldom (or a derivate there of) was used to parse XML in OpenACS. As this was too slow, tcldom was replaced by ns_xml. However, what appears to be the original tcldom based procedures to parse XML are still around in the following libraries:

- xml-0-sgml-procs.tcl
- xml-1-dom-procs.tcl
- xml-2-procs.tcl

None of the procedures defined in these libraries are being called from elsewhere in the OpenACS code base. Is it safe to assume that these 3 .tcl files can be removed?

/Bart

Collapse
2: Re: Outdated XML libraries? (response to 1)
Posted by Jorge Garcia on 03/06/03 01:02 AM
Bart,

Maybe a little off-topic here, but could you please send detailed instructions to compile, install and configure tDOM with Aolserver?

If it's possible with the Aolserver v3.3+ad13 which is used as a base system in the install docs.

Sure many people will be very grateful.
Me the first 😊

Thanks.

Collapse
3: tDOM and AOLServer (response to 2)
Posted by Ernie Ghiglione on 03/06/03 01:14 AM
Jorge,

Here are some postings at the tDOM group community that I found really useful to compile and invoke tDOM from AOLServer

Compile tDOM with AOLServer

http://groups.yahoo.com/group/tdom/message/308

Invoke tDOM from AOLServer

http://groups.yahoo.com/group/tdom/message/372

(Bart comments are also included on the postings)

I hope this helps,

Ernie

Collapse
4: Re: Outdated XML libraries? (response to 1)
Posted by Bart Teeuwisse on 03/06/03 02:39 AM
Another reason for retiring the XML libraries is that they use 'package require' quite a bit. As we all know AOLserver 4.0 will be the first version to actually support this common Tcl instruction.

I found another core library that uses 'package require': utilities-procs.tcl. It checks for the Trf package and if present uses it in base64 encode and decode procs.

I'm wondering if the Trf based procs in the base64 namespace are really needed. Doesn't AOLserver provide ns_uuencode and ns_uudecode commands?

I'm aware of the bug in ns_uuencode (fixed in AOLserver 4.0). For that reason the Tcl based base64 procs are usefull. Although I must say that the toolkit doesn't use the base64 procs (Trf or Tcl based) anywhere. Instead the email-handler package simply calls ns_uuencode without checking if it bug free. Acs-mail on the other hand checks for a broken ns_uuencode and provides its own version of base64::encode. (The copy is almost identical to Tcl based base64::encode proc.)

Wouldn't it make more sense to replace the base64 namespace altogether with a Tcl based implementation of ns_uuencode should ns_uuencode as provided by AOLserver be broken? All packages can then continue to use ns_uuencode regardless the AOLserver version.

/Bart

Collapse
5: Re: Outdated XML libraries? (response to 1)
Posted by Don Baccus on 03/06/03 03:24 AM
The APM actually referenced those Tcl XML procs in one special case, to report an error. Someone missed a call.  Fortunately since our XML files don't have errors we didn't notice because the Tcl libs weren't being loaded anyway :)

Peter Marklund copied that APM code though in acs-lang for 4.7 and it of course needs to be tweaked.

Other than that they can go.  I did remove the proc that loaded them since it was commented out anyway ...

The *right* thing to do, which I did in many places to make our stuff work with old and new versions of nsxml, is to go use the xml wrapper procs in acs-tcl/tcl (just look for ns_xml and you'll find them.)  This minimizes code change at the client level when we change versions/parsers/blah/blah.

Currently it checks for old/new libxml/nsxml pairings and makes things work for the caller.  It would be nice if we could do that for tDOM too, at first - the OpenACS core makes such simplistic use of XML that I would hope there's some chance of being able to do this?

tDOM we want for 4.7 of course ... are the AOLserver/zoran folks agreed that his AOLserver module can be found in the CVS tree with other standard AOLserver modules?

If I were told that tDOM can be had with a standard AOLserver 4.0 release without having to chase it all over the net then I'd say we could forget libxml support in 4.7 altogether but 4.0 is the first time in years that we've been able to say "just use the standard AOLserver release and standard modules from their download site and you're set" and I hate to toss that out for the core packages minimal use of XML ...

I realize that more complex use of XML by heavily XML-driven packages (like SCORM) will want to use tDOM regardless, I just want to minimize the install hassle for newbies and the upgrade hassle for those with existing sites.

So does anyone know what the status of tDOM is in regard to being officially associated with AOLserver?  Will we be able to point people to AOLserver 4.0 and get them all the pieces they need for an AOLserver install?

Collapse
6: Re: Outdated XML libraries? (response to 5)
Posted by Bart Teeuwisse on 03/06/03 03:44 AM
Don,

I've replaced all ns_xml calls in the xml_ wrapper procs with calls to tDOM and subsequently replaced all other calls to ns_xml with calls to the wrapper procs. There is no technical need to keep ns_xml around. The above mentioned outdated XML libraries can go too, as far as I can see.

I don't know about including the tDOM module with the other AOLserver modules. That is a question to be forwarded to Zoran et. al.

Please note that tDOM works will all AOLserver versions from 3.3+ad13 upwards. (Based on personal experience.) There is no reason to upgrade to AOLserver 4.0 on behalf of tDOM.

Worst case upgrade scenario would be to upgrade OpenACS, download, compile and install tDOM (from the AOLserver repository or elsewhere), replace nsxml with nstdom and restart the server.

With AOLserver 4.0 we can expect to see more and more regular Tcl packages making their way into OpenACS. Like tDOM they will be only losely tied to AOLserver (or not all) and it is to be expected that most of them will not be available from the AOLServer directory.

Wouldn't it be better to setup a resouce center/directory on OpenACS with links and instructions to the software components that (some) OpenACS packages depend upon?

/Bart

Collapse
7: Re: Outdated XML libraries? (response to 1)
Posted by Don Baccus on 03/06/03 07:09 PM
I think you misunderstand slightly ...

What we've been looking forward to is to being able to install the *core* and basic packages with a standard AOLserver release.  AOLserver 4.0 and OpenACS 4.6.1 will be the first combination that's out-of-the-box installable without having to track down ns_ modules here or elsewhere, grab our custom hacks to AOLserver, etc.

It would be nice not to lose that for 4.7.  People keep pleading with us to simplify the hoops they need to jump through in order to get a basic install working.  We want to be as newbie-friendly as possible.

For other packages that add fancy capabilities to OpenACS, sure, I have absolutely no expectation that they restrict themselves to modules that are part of the Blessed Set hosted at aolserver.com.  Among other things, folks who are getting deep enough into developing an OpenACS site that they'll be looking to use such packages should be beyond the "newbie" stage and no longer intimidated by having to jump through a couple of new hoops.

I think the best answer is to put our political weight behind getting Zoran's tDOM-based AOLserver module loaded up at aolserver.com before we release 4.7.  Maybe some of the folks who are more involved in the AOLserver community than I could enlighten us.

Thanks for doing the work to make everything call the xml wrapper procs, that's great.

Collapse
8: Re: Outdated XML libraries? (response to 4)
Posted by Jamie Rasmussen on 03/06/03 07:16 PM
Actually, the ns_uuencode patch from OpenACS has not made it into AOLserver 4 yet - I just submitted a patch for it today, #698741 on SourceForge.
Collapse
9: Re: Outdated XML libraries? (response to 7)
Posted by Jamie Rasmussen on 03/06/03 07:47 PM
nsfts is another module in a similar situation.  By the way, nsfts doesn't compile on Windows yet, more on that in another thread perhaps.

There are a few other modules like nsimap and nsreturnz that I would like to see moved to SourceForge, even though they aren't needed to install OpenACS.  Having the latest modules in one place keeps people from reinventing the wheel, makes installation much easier, publicizes some of the powerful plug-ins AOLserver has, and helps keep infrequently used modules like ino_chart from disappearing.  In the meantime, I think my list of AOLserver modules is currently the most comprehensive: http://empoweringminds.mle.ie/openacs/aolserver_modules

The AOLserver chat should be starting soon and I'll ask Zoran about importing nstdom - since tdom isn't hosted at SourceForge, I don't think it would help much to have nstdom hosted with AOLserver.  I'm sure there is a reasonable solution to this problem, but I don't know what it is yet.

Collapse
10: Re: Outdated XML libraries? (response to 1)
Posted by Don Baccus on 03/06/03 11:50 PM
OK, Jamie, thanks.

I suppose we could always build our own comprehensive source tarball of the pieces we like and that folks might need, maybe that's really the best solution after all.

Keep us posted as to what's happening, and many thanks for bird-dogging this.

Collapse
11: Re: Outdated XML libraries? (response to 10)
Posted by Jamie Rasmussen on 03/07/03 01:35 AM
I was confused about this, but Zoran clarified it for me - sorry if this was obvious to everyone else. 😊

There is no separate "nstdom" module, there is just tDOM.  Since tDOM is not primarily an AOLserver module it probably doesn't belong in the AOLserver CVS repository.  The AOLserver "module" part of tDOM is basically a trivial Ns_ModuleInit that gets included if you configure with --with-aolserver before building.  (Not necessary for AOLserver 4.)

We could do the comprehensive tarball thing as Don suggests and ask that it be added to the AOLserver file download section on SourceForge.  Or we could upload some "known good" tarballs of things like tDOM so new users can just grab all of the needed pieces from one place.  That way we would also take advantage of the SourceForge bandwidth and mirroring system.

In other AOLserver chat news, 4.0 beta 3 is coming soon, probably tomorrow, and in addition to several bugfixes, it is supposed to have the remaining AD13 i18n features.  (That code has been checked in, but I haven't looked at it yet.)  In addition, a first draft of a module author upgrade guide was announced, and we discussed issues related to cataloguing and cleaning up non-core modules of various types.  The complete text of the chat is at http://empoweringminds.mle.ie/openacs/chatlog/today.html

Collapse
12: Re: Outdated XML libraries? (response to 1)
Posted by Don Baccus on 03/07/03 05:49 PM
Jamie - do you know if they fixed the "hang on signal" issue?

I guess my fantasy of being able to tell people "download the latest AOLserver version, download the latest PG version, download the latest OpenACS kernel and core packages, and install" is not likely to come to pass after all.

It's too bad, I see this as being a small but not insignificant barrier to adoption.

Maybe Jamie's intermediate suggestion - the AOLserver.com host a large, inclusive tarball that includes extra pieces - is a reasonable approach.

Collapse
13: Re: Outdated XML libraries? (response to 12)
Posted by Jamie Rasmussen on 03/07/03 09:03 PM
If you mean SF #667651 - Hangs on shutdown if scheduled procs have run - no it hasn't been fixed yet.  I'll see if we can boost its priority.  http://sourceforge.net/tracker/index.php?func=detail&aid=667651&group_id=3152&atid=103152

I'd certainly like to see ready-to-go source tarballs and binary distributions, at least for the common platforms.  I've been meaning to set up a nightly AOLserver build for Windows, but just haven't had the time.  I wonder if we could make use of SourceForge's compile farm.  Don't give up on that fantasy yet!

Collapse
14: Re: Outdated XML libraries? (response to 1)
Posted by Tom Jackson on 03/07/03 09:19 PM

Jamie,

It used to be the case that when I used daemontools 'svc -t /service/name', a schedule proc would hang, the server would stop except for about 4 threads. When I used 'svc -k /service/name', the server would immediately restart.

Is this what Don is concerned will prevent his 'fantasy'? Or maybe it is the modules not being in one sf location. I was wondering could the missing modules be replaced with a simple makefile that would wget the required module, or maybe just print a message of where to look? Scott G. has some makefile code to generate a tagged release, maybe that is close to what is needed?

Collapse
15: Re: Outdated XML libraries? (response to 14)
Posted by Jeff Davis on 03/07/03 10:21 PM
Tom, what we want to do is be able to get back on the head of aolserver for OpenACS and to my mind (and to others I think), not shutting down cleanly on sigterm is a showstopper. The issue I have with it is that when the machine shuts down if aolserver hangs this way then oracle does not shut down either (until the badly behaved things are delivered thier sigkills -- which I think means a sigkill for oracle as well). kill -9 on your db server is not necessarily a great idea.

If it's possible to have aolserver release all the db handles first or something like that so oracle can shut down I would consider it an unfortunate bug but not a show stopper.

Collapse
16: Re: Outdated XML libraries? (response to 1)
Posted by Tom Jackson on 03/07/03 10:51 PM

Jeff,

So how are the webservers being shut down? I am now using an init script to do this as the first step in server shutdown/restart. I was forced to address this issue because of Oracle and sshd. In RH8.0 sshd would shut down first, then Oracle would try to shut down. However since the webservers were still connected to Oracle, it would just sit there. In the past I would have to login via ssh and shutdown the webserver, thus allowing Oracle to stop. But since sshd was now shutting down first, I had to muck around with the init scripts.

That is when I realized that there was essentially no order in the shutdown of AOLserver, just relying on the server to kill everything. So now my first step is to run 'svc -d /service/*'. I'm not sure what the signals are in svc -d, but I haven't had a problem doing this. Maybe I should run a simple test and report back.

Anyway, that is the way I do it. I'm not sure how Don or you shutdown AOLserver during a reboot.

Collapse
17: Re: Outdated XML libraries? (response to 16)
Posted by Jeff Davis on 03/07/03 11:33 PM
Tom, I am starting aolserver4 from daemontools and it hangs on any signal other than sigkill. -d sends sigterm then sigcont neither of which seem to do the trick. It also hangs when run in the foreground and killed with sigint and sigterm.

I have run it in the debugger and it gets the signal just fine and starts shutdown but hangs trying to clean up one of the sched threads.

Collapse
18: Re: Outdated XML libraries? (response to 1)
Posted by Tom Jackson on 03/08/03 11:41 PM

Jeff, this is new behavior. The test case I ran was as follows: 


proc sleepy { } {

    ns_log Notice "Sleeping for 120 seconds."
    ns_sleep 120
    ns_log Notice "Finished sleeping."

}

ns_schedule_proc -thread -once 30 sleepy

When I watch the log, I see the first log notice and used: 'svc -d'. The sched thread kicks into action says 'waiting for event threads...'. My scheduled proc sleeps for 120 seconds an d then prints the second log notice. But then sched keeps on waiting. Even when the event thread is finished, it still hangs.

What did work was to do this: 'svc -k; svc -d', so the heavy handed signal kills everything, then, really before the server has time to shutdown, it gets the signals from 'svc -d' and stays down. It works the other way around as well, but the log doesn't contain the access log message from -shutdown- and the exiting message from -main-.

Collapse
19: Re: Outdated XML libraries? (response to 1)
Posted by Roberto Mello on 03/09/03 07:35 PM
Neophytos had AOLserver running with tDOM instead of ns_xml a long time ago. He sent me his modifications. I'll post them to my website (url coming) so people can see the diff's.

-Roberto

Collapse
20: Re: Outdated XML libraries? (response to 19)
Posted by Bart Teeuwisse on 03/09/03 10:30 PM
Roberto,

It would be interesting to see Neophytos's diffs. I recently swapped ns_xml for tDOM and started to clean up the OpenACS code, which is how this thread started. Chances are that my modifications include those of Neophytos. I would love to compare notes with his approach.

When would be a good point to commit these changes to CVS? Remember this will break backwards compatability unless we explicitly support tDOM and ns_xml at the same time. Do we care about support for ns_xml when tDOM has become the prefered XML library? If so how long should both libraries be supported? More food for thought...

/Bart