Forum OpenACS Q&A: Re: AOLserver 4.0 Install instructions
At that time i started to look more detailed into the problem, and found out it happened when all connection threads became busy and still more requests are queued. The problem was that every connection thread after the other went into a busy loop until reaching a complete freeze if the server. After running fine for a while, first only one connection thread went under high load into a busy loop (load 1, cpu 100%) while the server was still responding. Some time later, a second and third connection thread did the same (e.g. load 3), making the resource situation worse. Finally, when all connection threads went busy, the server hangs and stops replying.
Already in jan 2006 Jeff Rogers sent a patch for aolserver 4.0 to the aolserver list, dealing with a problem he was running into when benchmarking aolserver/openacs with ab (apache benchmarks). While benchmarking openacs aolserver run out of resources and reached a hang. Although he reported different symptoms (no busy cpu) i "ported" the patch to aolserver 4.5 . Since the patch is applied on openacs.org (mid of dec 2006), no hangs appeared again.
The patch was accepted for aolserver 4.0 and 4.5. For more details, see http://sourceforge.net/tracker/index.php?func=detail&group_id=3152&atid=103152&aid=1615787
Tom, are you asking out of academic interest, or do you have problems with the patch, or are you fighting similar symptoms?
This is not academic at all. I see the registration of the callback in driver.c. But in the source code, I don't ever see the callback called. Somewhere NsRunAtReadyProcs must be called, or the TriggerDriver signal will never execute. I can't find this, either because it isn't there, or because my checkout is wrong or I can't use grep. I'm okay with any of these, but simply registering a callback and never using it seems unlikely to solve a real problem.
Also, I have no problems with the patch, unless it isn't actually being used by anyone. There are huge differences between 4.0x and 4.5 wrt this subject so I don't assume that the solution would be so easily moved to 4.5.
The code is relatively undocumented. I'm an expert in undocumented code. A few months or years later I wonder why things are like they are. Why did I do that? So if the code is doing something, I'd like to make a note so it isn't removed. Tricky stuff is worth noting.
If you have time to investigate further, i would suggest to take out the patch and try to reproduce the bug in a clean-house environment. It seems related to threads with a larger footprint (nobody saw it except openacs applications).
The fact that this helped out somehow (in a big way) is important to know.
I also noticed that NsRuAtReadyProcs() is extern, but it is only defined in nsd.h, not include/ns.h. This means, as I just learned, that it can't be in a module, it has to be compiled into libnsd. Btw, the only place I found any *Procs() called was in nsmain.c.
The registered callback is TriggerDriver, and the use of TriggerDriver has changed. The similar SockTrigger used to be conditional during Sock Close, but now TriggerDriver gets called anytime there is mutex unlock on the driver pointer structure (except in ns_driver query, where it is called inside the mutex). Maybe an important one is the fact that TriggerDriver gets called if, from Tcl, you query the driver! So some new Tcl diagnostic code could unstick the driver thread, maybe like a scheduled proc.