Forum OpenACS Q&A: ns_atexit / ns_atshutdown failing silently

when I do the following (in my library, or from nscp)

ns_atshutdown ns_log {notice "at shutdown"}

ns_atexit ns_log {notice "at exit"}

neither gets executed.  no error messages are generated either.  I'm
running nsd 3.3+ad13.  any ideas?

Collapse
Posted by David Walker on
Try
ns_atshutdown {ns_log notice "at shutdown"}
Collapse
Posted by Jonathan Ellis on
same deal; nothing.  I've also tried scheduling procs with no args that simply do an ns_log; nothing there either.  I couldn't find anything in nsd.tcl that says "ignore ns_atshutdown commands" either. :0
Collapse
Posted by David Walker on
How are you shutting down your server?  nsd -K and ns_shutdown should
trigger these commands.  kill -9 nsd probably won't.

I tested with nsd -K and it worked fine for me.  3.3.1+ad13/nsd8

Collapse
Posted by Jonathan Ellis on
I tried both with SIGTERM and -K, which looked like

/home/aolserver/bin/nsd -s bf2 -Kt /home/aolserver/sfr.tcl -u nsadmin -g web

this is exactly the line I use in inittab to run it except w/o the -i, and with the K, which I assume is the correct way to use that.  running same version of nsd as you.  I tried running with debug=true and didn't see anything besides the usual

nssock: triggering shutdown

stuff.

Collapse
Posted by Jonathan Ellis on
I did get it to work by running nsd -f and control-Cing it.  Now to figure out how to -K correctly...
Collapse
Posted by Jonathan Ellis on
in my test server -K and TERM both work fine. here's what it says:
Notice: nsmain: AOLserver/3.3.1+ad13 stopping
Notice: nssock: triggering shutdown
Notice: serv: stopping connection threads
Notice: serv: connection threads stopped
Notice: sched: shutdown pending
Notice: sched: shutdown started
Notice: sched: shutdown complete
Notice: exiting
Notice: drv: driver 'nsunix' stopping
Notice: socks: shutdown pending
Notice: nscp: shutdown
Notice: socks: shutdown complete
Notice: at shutdown *** MY NOTICE
Notice: nslog: closing '/home/aolserver/log/sfr/sfr_access.log'
Notice: nssock: shutdown complete
Notice: nsmain: AOLserver/3.3.1+ad13 exiting
but on my production server I only get
Notice: nsmain: AOLserver/3.3.1+ad13 stopping
Notice: nssock: triggering shutdown
Notice: drv: driver 'nsunix' stopping
Notice: keepalive: shutdown pending
Notice: keepalive: shutdown complete
Notice: serv: stopping connection threads
The difference appears to be that my production server is starting a keepalive thread even though I have keepalivetimeout = 0 specified which is supposed to disable it. The test server ran off the same config file and did NOT start a keepalive thread. I noticed that Ns_ConnClose always calls NsKeepAlive on to-be-closed connections so figured maybe the difference was I never actually requested any pages from the test server. So I did that; no difference.

So now my second question, probably related to my first, is, how do I REALLY kill keepalive?

Collapse
Posted by Jonathan Ellis on
to answer my own question, you REALLY kill keepalive by specifying keepalivetimeout = 0 in the correct section, which is ns/parameters.

but that turns out not to be why my ns_atshutdown isn't happening. :(

Collapse
Posted by Jonathan Ellis on
I ripped out jerry's vhr patches, and still no love from the shutdown procedures.  I didn't think that would be the problem, since both test and production servers were running those same patches, but I tried it anyway.

Note that it's not just "my" ns_atshutdown that doesn't work; under the hood, ns_atshutdown calls Ns_AtShutdown (surprise :0) which is the same way nssock schedules it's cleanup proc that logs the "nssock: shutdown complete" message.

The only difference between test server and production is that production is under heavy load.  So I fired up my test server again, jammed ab -100000 down its throat, and hit it with a SIGTERM.

It didn't run the atshutdown procs.  I tried it twice more both ways to be sure and it's consistent: under load, no ns_atshutdown.  No load, ns_atshutdown runs fine.

I'm disappointed how the more I push the limits of nsd the less robust it appears.  I could roll my own "shutdown" script that signals a filter to stop processing requests, runs any scheduled procs, then kill -9s itself, and I guess that's what I'll end up doing.  Anyone have any luck with nsd 3.4? :P

Collapse
Posted by Jonathan Ellis on
the other fun thing is that ns_shutdown works by sending nsd a SIGTERM which can cause things to hang (looping endlessly, from the looks of top) after it logs "exiting".  I've written replacements which I'll post separately.