I'm reposting this here for the benefit of anyone who might not read the AOLserver list.
We had analyzed our server logs and just couldn't come up with a pattern. Today, however, I bit the bullet and am running the production server under gdb. It is consistently crashing in the PayFlowPro module.
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 9226 (LWP 5320)]
0x409ae28c in __umoddi3 ()
from /usr/local/verisign/payflowpro/linux/lib/libpfpro.so
(it sometimes crashes elsewhere in the library, often in pfproVersion, but the segfault is always in libpfpro.so)
I found a reference to this on Google:
http://groups.google.com/groups?q=sigsegv+__umoddi3+payflowpro&hl=en&lr=&ie=UTF-8&selm=aachc7%24kiu%241%40FreeBSD.csie.NCTU.edu.tw&rnum=2
Which basically says the that payflowpro library is statically linked with OpenSSL 0.9.5, but we are running 0.9.6c. The PHP folks have this problem too.
This explains why our dev servers didn't crash with this problem--the pfpro module just doesn't get as much stress there as it does on production.
Now, how to fix this? Not sure, we are trying to get a beta version of the lib from Verisign right now.
