Forum OpenACS Q&A: ad_schedule_proc seems to be failing

Hi all, I seem to have a serious problem since a server restart of a server I work with on the weekend (13/6/05) - during startup, although ad_schedule_proc is called to queue up several repeatedly running scheduled methods, they're no longer being called anymore.

The system is running Linux 2.4.2-2, with ACS AOLserver/3.2+ad12 (I know it's not OpenACS, but I'm hoping I can nevertheless get some help).

I've traced through the ad_schedule_proc code, and couldn't see any problems, with it passing the request onto ns_schedule_proc. The strange thing is that when I changed the date back several months, the server restarted fine again, running the scheduled procs smoothly.

I'm kind of worried that I may have fallen victim to some kind of integer overflow problem with the representation of date/times as integers (that's the only thing I can think of), and I'm pretty much in over my head. I'm hoping someone can offer some suggestions.

Thanks.

I seem to be seeing this as well, on our last remaining 3.3+ad13 installation. Scheduled procs aren't firing anymore.

Unfortunately upgrading is a last, last resort (client's wishes) so I need to try to figure this out I guess. Very, very strange.

I'll go post to the AOLserver list also.

If this is a "known" problem, what's known about it? Something in AOLServer? Some OS problem? Some weird date-related problem? Something entirely unrelated to OpenACS? Where has this been discussed before? The discussion over on the AOLServer list by people who know vastly more than I doesn't allude to any prior knowledge of this (http://news.gmane.org/gmane.comp.web.aolserver)

Seems pretty strange for production sites to suddenly develop problems after running fine for years -- and for other sites apparently not to manifest problems. I've restarted clones of my OpenACS 3.2.5/AOLserver/3.3.1+ad13 sites on a dev box and found no problems with VM or with scheduled procs not firing. I'm not restarting the production sites to see what happens until I learn more about what might be going on.

This is way further into the plumbing than I enjoy venturing, so I certainly hope those who can peer deeply into the stack will let us know what is afoot.

Other than the VM and scheduled proc problems, has anyone had any other difficulties with AOLserver/3.3.1+ad13? Specifically, has anyone seen problems with Jerry Asher's nsvhr/nsunix virtual hosting method? Is anyone else even still using Jerry Asher's nsvhr/nsunix virtual hosting method?

I am also using AOLserver/3.3.1+ad13 with Jerry Asher's nsvhr and nsunix.

All seemed to be working well, until yesterday. After restarting a server instance today (18 May 2006) the nsd server starts listening on the appropriate socket but then shuts down with:

nsthread(19614) error: pthread_create failed in NsThreadCreate: Cannot allocate memory

I have seen the same error on more than one system. No changes have been made at the OS or database levels prior to these errors cropping up.

Does anybody have an idea how to fix or work around this error? Does it mean upgrading to AOLServer 4? I am experiencing these errors on production sites, so would like to get them back up and running ASAP. I'm not having any problems with AOLServer 4.0.10.

Last few lines of error log:
-----------------------------
[18/May/2006:12:19:56][19614.1076616112][-sched-] Notice: sched: starting
[18/May/2006:12:19:57][19614.1074158336][-main-] Notice: serv: warmed up
[18/May/2006:12:19:57][19614.1074158336][-main-] Notice: socks: idle
[18/May/2006:12:19:57][19614.1074158336][-main-] Notice: sched: idle
[18/May/2006:12:19:57][19614.1074158336][-main-] Notice: nsunix: DrvStart starting: listenSocket = 9, modules/nsunix/jdio.nsunix
[18/May/2006:12:19:57][19614.1074158336][-main-] Notice: nssock: listening on 0.0.0.0:8010
[18/May/2006:12:19:57][19614.1138777008][-nssock-] Notice: nssock: starting
[18/May/2006:12:19:57][19614.1136671664][-thread1136671664-] Notice: nsunix: accepting
[18/May/2006:12:19:57][19614.1138777008][-nssock-] Notice: nssock: accepting connections
nsthread(19614) error: pthread_create failed in NsThreadCreate: Cannot allocate memory

But what is the best way to handle virtual hosting for multiple sites on the same server box?

Hey Stan,

AOLserver 4 does have virtual hosting built-in but pound is more flexible. You can stop/start individual servers and it gives you the option of running non-AOLserver web servers if you want 😊

Built-in AOLserver virtual hosting instructions are here: http://panoptic.com/wiki/aolserver/Virtual_Hosting

If you want to use pound, it's pretty simple to set up. My /etc/pound/pound.cfg file looks something like this:

...

ListenHTTP
  Address     66.98.222.124
  Port        80
  xHTTP       0
  WebDAV      0
  Change30x   1
End

Service
  HeadRequire "Host:.*kurup.org.*"
  # make sure not to match other servers
  HeadDeny    "Host:.*dev.kurup.org.*"
  BackEnd
    Address   127.0.0.1
    Port      8001
  End
  Session
    Type      COOKIE
    ID        "ad_session_id"
    TTL       1800
  End
End
...

Also, if you want to have different domains and have ssl on each one, AOLServer with virtual hosting will not handle it.

As I understand it, the domain name in the http request will be encrypted, so AOLServer will not know which ssl certificate to present.

Pound can help you do this as well (haven't tried it but so I've read)

Since no one else has posted it here yet.

'Jesus' Jeff Rogers has solved the Problem and posted a solution on the AOLServer mailing list.

It's amusingly enough a y2038 problem.

http://news.gmane.org/gmane.comp.web.aolserver/

"I fixed it by simply changing my MaxOpen/MaxIdle settings to "0" which is interpreted as "forever" which is probably what the original "one BILLION seconds" was undoubtedly intended to be."

I don't fully understand the effects of this problem. If it's an Oracle driver problem, shouldn't it affect all AOLserver versions, not just 3.3+ad13? Does it only affect 32-bit operating systems? One particular client of mine has recently been having a lot of problems with scheduled procs not always running - I wonder is this issue an element? They're seeing the problem on both AOLserver 3.3+ad13 and 4.0.10 (64 bit).

thanks
Brian

Hi Stan,

yes I've set MaxOpen and MaxIdle to 0 on all my systems. I was just trying to understand the problem more clearly - I haven't really yet seen an explanation I can understand. And I'm not even sure if the problems my client has been seeing is related to this year 2038 issue.

Unfortunately, I've been unable to mail the AOLserver list due to a reverse DNS issue we're having here, but I have my mail ready to go once the DNS is sorted.

thanks
Brian

Thanks Michael! Very clear explanation.

Brian