Forum OpenACS Q&A: AOLserver dying unexpectedly

Greetings,

We have a brand-new 4.6.2 site up and running.  AOLserver is dying unexpectedly at random intervals.  No messages in syslog or the server log, no indication that a certain URL is causing it.  Machine has 4 GB of RAM, is not swapping, and hosts 4 other AOLserver instances that have uptimes in the months.  Debian 3.0, kernel 2.4.20.  We've tried it under AOLserver 3.3+ad13 and AOLserver 3.5.6.

At Jon Griffin's suggestion we made sure that custom ErrorDocuments were defined and are being served.

Since I first tried to post this we have discovered that our old site (ACS 3.4.x based, aol33+ad13) also does the same thing, just not as frequently.  It has never done that before, it has run for months on end.

I am baffled.  Any suggestions?  Stacksize setting? (currently 500000)

One truly bizzare thing I noticed is *very* large values of /proc/sys/fs/file-max (line in the 300000+ range).  On our development server that number never goes about the set value of 10240 (which I put there for Oracle).

Any suggestions are welcome.

Are you running Redhat?  There was a RH version in the 7s, I've forgotten which one, which changed something that required everyone to up their stack size.  I currently have "expr 1024*1034" for the main stacksize (in ns/parameters) and "expr 512*1024" in the per-thread stacksize.

I'd try this even if you're not on Redhat, because the result of a too-small stacksize is exactly what you describe, sudden and immediate nsd death.

Typo, sorry.

I'm reposting this here for the benefit of anyone who might not read the AOLserver list.

We had analyzed our server logs and just couldn't come up with a pattern.  Today, however, I bit the bullet and am running the production server under gdb.  It is consistently crashing in the PayFlowPro module.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 9226 (LWP 5320)]
0x409ae28c in __umoddi3 ()
  from /usr/local/verisign/payflowpro/linux/lib/libpfpro.so

(it sometimes crashes elsewhere in the library, often in pfproVersion, but the segfault is always in libpfpro.so)

I found a reference to this on Google:

http://groups.google.com/groups?q=sigsegv+__umoddi3+payflowpro&hl=en&lr=&ie=UTF-8&selm=aachc7%24kiu%241%40FreeBSD.csie.NCTU.edu.tw&rnum=2

Which basically says the that payflowpro library is statically linked with OpenSSL 0.9.5, but we are running 0.9.6c.  The PHP folks have this problem too.

This explains why our dev servers didn't crash with this problem--the pfpro module just doesn't get as much stress there as it does on production.

Now, how to fix this?  Not sure, we are trying to get a beta version of the lib from Verisign right now.

Last night we reversed the module load order in nsd.tcl so the verisign module (which subsequently loads the PayFlowPro libraries) is loaded *after* nsopenssl.  Our server has been up for almost 24 hours, a record.  I think that is at least a working solution.  The real fix, of course, would be to get a copy of libpfpro.so dynamically linked against OpenSSL, but Verisign still has not returned our calls to level 2 technical support.

So for the record, it appears fixed. 😊

Thanks to everyone that responded here and on the AOLserver list.  As I said over there, helpful people like you are what makes open source work.

I forgot to mention earlier that our server does still crash, it just does it much less frequently now.  Since it can take up to 90 seconds for it to auto-restart, we've decided (as Brad mentioned above) that Verisign is no longer an option.