Forum OpenACS Q&A: Can someone recommend a good way to watch the server?

Collapse
1: Can someone recommend a good way to watch the server?
Posted by Nima Mazloumi on 02/04/06 07:43 AM
Hi,

since our server crashes every once in a while I need a simple but effective way to watch the server and restart it if it crashed and/or restart it after 24h. Has someone implemented something like this via cron job or something similar and could he/she post what and how to do that?

Greetings,
Nima

Collapse
2: Re: Can someone recommend a good way to watch the server? (response to 1)
Posted by Malte Sussdorff on 02/04/06 08:00 AM
/web/yourserver/etc/keepalive-cron.sh, run every 5 minutes from the crontab.

Steps:

  1. Edit /web/yourserver/etc//keepalive/keepalive-config.tcl by adding
    set servers_to_monitor {http://yoursite.de "/usr/local/bin/svc -dku /service/yoursite"}
  2. Edit your crontab with "crontab -e" and include the script to run every 5 minutes.
    */5 * * * * /web/yoursite/etc/keepalive/keepalive-cron.sh

Works like a charm.

Collapse
3: Re: Can someone recommend a good way to watch the server? (response to 1)
Posted by Nima Mazloumi on 02/04/06 08:25 AM
Looking at the scripts the cron job basically makes a http request and if the server doesn't respond restarts it.

But what I need is
- to check if server has crashed and restart it
- to restart server after a given period

I will try to extend this script.

Do I need daemontools for this?

Collapse
4: Re: Re: Can someone recommend a good way to watch the server? (response to 3)
Posted by Malte Sussdorff on 02/04/06 08:34 AM
Hi Nima,

what is the significant difference in your opinion of not responding and crashing? A server that is not responding needs to be restarted.

If you use daemontools (which is required and anyone not using it or something alike is digging their own grave IMO), a crashed server will be restarted automatically, though this wont help you if the server is continuously running, but not responding (for whatever the reason). Thats what keepalive is for.

If you want to restart the server every morning at 4am, write into the crontab "* 4 * * * svc -dku /service/yoursite" (maybe there is a start missing.).

If you want to restart it after not responding for some time edit /keepalive-cron.sh and play with the wget parameters.

Collapse
5: Re: Can someone recommend a good way to watch the server? (response to 1)
Posted by Carl Robert Blesius on 02/04/06 09:48 AM
Daemontools will do this for you. You can set the time to wait before restart with "sleep" within the run script. I think the default is 10 seconds.

Works well.

Installation is documented here:

https://openacs.org/doc/current/install-daemontools.html

Collapse
6: Re: Can someone recommend a good way to watch the server? (response to 1)
Posted by Andrew Piskorski on 02/04/06 11:22 AM
Restarting AOLserver when it crashes is trivial, just run it from /etc/inittab. Or use Daemontools, which is a different and more modern way to accomplish basically the same thing. inittab is simpler, and all Unix systems have it. Daemontools is more flexible.

Crontab is of course a good way to schedule a periodic keep-alive job to check that your AOLserver is not wedged. If AOLserver is wedged, the keepalive script merely kills AOLserver, which init or Deemontools then restarts.

Isn't this all in the OpenACS install docs? I thought it was, and hope it is.

Periodically restarting an AOLserver which is performing normally shouldn't really necessary, but if you want to do it just call ns_shutdown from within AOLserver.