Forum OpenACS Development: Two Technical Questions

Collapse
Posted by Maurizio Martignano on
Dear members of this community I'd like to hear your expert opinion on two technical topics concerning Aolserver/OpenACS. It might be that these questions derive from me being quite a newbie on this domain, so it might be you already have an answer/a solution handy.

1. Memory Allocation in Aolserver/TCL
As far as I understand it seems to me that Aolserver uses (for the execution of TCL scripts) the memory allocator of TCL itself.
Depending on how TCL is compiled (USE_THREAD_ALLOC, etc….) there are three (as far as I know) possible variations/versions of this memory allocator:
1. Standard (the one using the standard malloc and free of the underlying operating system; this is the one consuming less memory)
2. Zippy (the default one, put in place by the option –enable-threads, pretty fast but not so good in saving memory)
3. VTMalloc (this is the allocator used inside Naviserver, a solution somehow in between the first two, not so hungry of resources but still quite efficient).
None of these allocators performs any sort of (real) garbage collection so memory fragmentation is an issue with all of them.
We have the need to develop a Web application able to run on a 24/24 and 7/7 basis (we cannot afford a nightly restart of the server).
So my plan at the moment would be:
1. To use the less memory consuming memory allocator (i.e. # 1)
2. To study/measures the actual behavior of my application under realistic stress conditions and derive from that some sensible values for the parameters in the config.tcl.
While this might be a sort of workaround, a solution valid in the short term, I was thinking about something better for the long run. How about adding real garbage collection capability to the TCL memory allocator (I was thinking about the “Boehm Conservative Garbage Collector”) http://www.hpl.hp.com/personal/Hans_Boehm/gc/ .
Any thoughts on this?

2. Development Environment
I work for an operation where we have several developers working on .NET, Java and OpenACS/TCL. Quite often we have the need to move people from a project to another one, that means from a technology to another one. If possible I would like to minimize the stress for these people by reducing the amount of changes they have to go through whenever they move from an assignment to the next one. Our developers are more and more familiar with Eclipse. I noticed (and also used/tested) a standard plugin for Eclipse called DLTK http://www.eclipse.org/dltk/ .
This looks very interesting to me and would offer a solution to the migration problem. Has anyone of you ever considered to drop the use of Emacs in favour of Eclipse+DLTK?

Thanks a lot in advance,
Maurizio

Collapse
Posted by Malte Sussdorff on
If you get the memory problems working this would be great, but my experience is that most web sites that run into this problem actually have the need to run in a cluster environment anyway. And there a nightly restart does not have an impact.

We are using the VTMalloc which works nicely.

With regards to Eclipse: Yes, I though about it and posted for the google summer of source (use https://openacs.org/xowiki/GSoC?revision%5fid=905352 as it was edited in the current version). I did not look at DLTK though. If you have something going there, I am more than happy to act as a tester and collaborate on functional requirements. You might also want to get Nima on board as he has extensive knowledge in using Eclipse (if I recall correctly).

Collapse
Posted by Maurizio Martignano on
Dear Malte,
Eclipse+DLTK (Dynamic Languages Tool Kit) includes support for TCL. So I think it can either be used as it is to develop TCL scripts or used as starting point for the implementation of all the features planned in your previous post on OpenACS support inside Eclipse. At the moment I would say that Eclipse+DLTK plus some SSH based file transfer mechanism (e.g. WinSCP) to exchange files between the development and target machines could constitute a very effective workbench.

Cheers,
Maurizio

Collapse
4: AOLserver and malloc/free (response to 1)
Posted by Andrew Piskorski on
Maurizio, you haven't really said what you want to use AOLserver for. What is your use case? Evalauating the technical merits of different esoteric approaches isn't very useful without that.

I think (could be wrong though) that whether a memory allocator is subject to fragmentation problems or not has little or nothing to do with whether or not the system uses garbage collection. It also doesn't necessarily have much to do with whether the applications memory use continues to grow over time either. Remember that the operating system malloc does not necessarily release memory from the process just because you call free to release it! (And that varies across platforms.) That's one of the things Zoran and the other Naviserver guys wrestled with in their VTMalloc work, I believe.

Also, I suspect that retrofitting a Boehm-Demers-Weiser conservative garbage collector to Tcl and AOLserver, while certainly doable, is unlikely to achieve your goals. "Conservative" garbage collectors are just that, conservative - they lack full knowledge of the application's memory use, and when unsure, leave memory uncollected. Thus, all conservative collectors leak to some degree over time. Whether or not that's a problem is a matter of use case and of degree, but it's why conservative collectors are reputed do be unsuitable for genuinely long running, high use, high reliability servers.

AFAIK, dropping a Boehm-Demers-Weiser garbage collector into existing C code is primarily a way to avoid having to fix your malloc/free bugs and/or to have code written in a garbage collected language interact well with C code, not a way to get perfectly leak free code. (Garbage collection is a truly wonderful feature, but it's no magic wand...) I suppose both using a conservative collector and carefully calling free() might work better than either alone - dunno, I have no practical experience there.

For a super reliable leak free long-running server, you want either an exact garbage collector (which requires language support, and thus is more complicated and cannot simply be dropped into existing C code), or you want what AOLserver already has - mannual malloc and free. And then track down the few remaining tricky leaks you've missed...

Collapse
Posted by Maurizio Martignano on
Dear Andrew, thanks a lot for your reply. I'm afraid the memory leak or, if you want, the continuos growth of the memory occupied by the combination Aolserver/OpenACS does not depend on my particular application. Just install OpenACS, move around some few pages (try for instance the included test pages) and have a look at the memory occupation.... you'll see that it is growing. So this is an endemic behaviour of the infrastructure and not of the particular applications put on top of it.

Cheers and thanks,
Maurizio