Forum OpenACS Q&A: AOLserver 4 and daemontools

Collapse
Posted by mark dalrymple on
Greetings,

I'm having difficulty running AOLserver 4 with daemontools. It's very strange, breaking as soon as I add the -b switch to tell it to bind to a particular port. The 'run' script runs perfectly if I run it at the command line, and it runs ok with this as my runfile:

#!/bin/sh
exec /usr/local/aolserver40r2/bin/nsd -it /web/front-end/front-end-as4.tcl -u nsadmin -g web
My front-end-as4.tcl file has it bind to my IP address and port 8080. It starts up fine and handles requests (it's just a TCP nsvhr, no postgres driver or anything fancy like that).

If I use:

#!/bin/sh
exec /usr/local/aolserver40r2/bin/nsd -b 66.98.193.240:8080 -it /web/front-end/front-end-as4.tcl -u nsadmin -g web

Then it doesn't work (and I've tried moving the -b IP:port to different parts of the command line. % sudo svstat /service/front-end/ shows that it's continually trying to start something. Doing a ps and grepping for readproctitle doesn't show any error messages. I can't seem to find a log file where daemon tools is putting stuff. When I do a sudo ./run (either version) it works fine.

Thanks in advance for any assist.

Collapse
Posted by Lachlan Myers on
Very odd. Here's what works for me - service0 used to protect the guilty :)
#!/bin/sh

exec /usr/local/aolserver/bin/nsd-postgres -it var/lib/aolserver/service0/etc/
service0.tcl -u lach -g web -b service0.net:80,service0.net:443
This should NOT have line breaks in the exec call - I have noticed that its sensitive to *any* line breaks. Using IP dot quad shouldn't cause a problem? The nsd-postgres just sets up the postgres env variables before calling nsd, so that shouldn't change things. I'm using aolserver 4.0 not rc anything.
Collapse
Posted by Tom Jackson on

Could be a permission issue, since it starts up fine as root, or maybe a mismatch between the -b switch and what is in the config file.

OTHO, how did you start/stop it with daemontools? I didn't catch that part.

Collapse
Posted by mark dalrymple on
> OTHO, how did you start/stop it with daemontools? I didn't
> catch that part.

What I have been doing is copying my aolserver4 run file into run, then doing svc -t /service/front-end, which stops the aolserver3 version of the file. I also tried doing an svc -d, waiting for a handful of seconds, then do a svc -u. Also putting in a sleep 30 into run didn't help.

> or maybe a mismatch between the -b switch and what is in the config file.

they're the same. Also tried using the hostname instead of an IP address.

Luckily my web sites are nsvhr'd and are on high ports (don't need -b) and front-end is just my reverse proxy, so I'm not really blocked from getting useful stuff done, but it'd be nice having the same aolserver version for everything.

Wheeee! we having fun yet? 😊

Collapse
Posted by Tom Jackson on

Well one thing: where is the AOLserver log file? svc isn't going to have a log file, unless you set one up, and you need to run AOLserver with -ft to get that to work. An error need to show up somewhere. Even if the -b switch was incorrect AOLserver would still start up just fine, it just wouldn't listen.

Another thing: use -k to kill AOLserver unless you know -t works. If you want it to stop, do a -k and then a -d. You can also test by starting with a -o (once). Either the AOLserver error log or readproctitle will have an error for a restart, unless you have discovered a bug in nsmain. Don't know what would happen then.

I would report it to the AOLserver list, maybe someone else has seen similar behavior.

One more thing: if you are using a high port, why not login as nsadmin and start it up without sudo (./run, and maybe with -ft). Just hoping to expose an error somehow.

Collapse
Posted by mark dalrymple on
> Well one thing: where is the AOLserver log file?

It's in /web/front-end/logs/front-end-error.log. I have a tail -f on that file. When I restart with AS4 plus the -b flag, nothing appears in that file.

> if you are using a high port, why not login as nsadmin and start it up without sudo (./run, and maybe with -ft).

It works fine 😟, stuff appearing in the -error log and it handles requests.

> Either the AOLserver error log or readproctitle will have an error for a restart,
> unless you have discovered a bug in nsmain.

The -error never gets anything written into it when using -b (which seems totally bizarre), without the -b stuff goes to the aolserver log. grepping for readproctitle never shows anything.

Guess it's time to dig into the aolserver source and see what's different in -b land.

Collapse
Posted by Randy O'Meara on
Mark,

Deamontools must run aolserver in foreground mode (-f) not inittab mode (-i). Here's my run file.

#!/bin/sh

IP=192.168.1.250
port=8001
server=s${port}
serverroot=/devel/${server}

export AOLSERVER_HOME=/opt/aolserver4
exec 2>&1
exec $AOLSERVER_HOME/bin/nsd -fzt \
	${serverroot}/aolconf.tcl \
	-u ${server} -g ${server} \
	-b ${IP}:${port}
Collapse
Posted by Tom Jackson on

Well the error has to be in nsmain I think, since errors are not written to the error log until after all command line params are written and until the config file is read without error. I guess -b does mean bizarre. You could also do a little testing with the -B switch to see what happens. I think it takes the path to a file containing the ip:ports one per line.

Collapse
Posted by Bart Teeuwisse on
For the record, nsd does NOT have to run in foreground mode when using daemontools. ONLY when using multilog DOES nsd have to run in foreground mode so that Daemontools can pipe the output of the supervised process (nsd) to multilog.

The standard logging will go to never-never land if you start nsd in foreground mode using daemontools but don't use multilog.

As for the problem at hand, I would suggest switching to foreground mode and then start the run script from the command line: /service/front-end/run.

This will print all log messages to STDOUT and gives you a change to examine the log.

I can also advise the use of multilog in combination w/ daemontools. Your error logs will roll when they get too large and you get specify how many old log files you like to keep around. See https://openacs.org/forums/message-view?message_id=120023 for instructions how to use multilog w/ AOLserver.

I'm not sure what the problem is here. I'm running AOLserver 4.x just fine under daemontools. It might be a permission issue.

/Bart

Collapse
Posted by Randy O'Meara on
Bart's suggestion is a good one. What do you see when running it with a command equivalent to the run file I provided?

I've seen odd logging behavior due to permissions.

You could try this: (modify commands to match your OS utilities)

$ su - root
# cd /web/front-end
# svc -d .
# svstat .  (make sure it's down)
# chown -R nsadmin.web supervise log   (assuming "log" is correct logging dir)
# su - nsadmin
$ cd /web/front-end
$ svc -u .
$ svstat .  (make sure it's up)
$ ps -axw | grep readproc
/R
Collapse
Posted by Mark Aufflick on
More datapoints,

I have aolserver 4 running on both port 80 and 443 (ssl) using the -B flag and it starts and runs fine. Why you can't just use multiple -b options, though, makes no sense to me.

I don't use multilog, and I use the standard run script (plus the -B option).

The only problem I have is with restarting. cvs -t restarts fine, except the server complains it can't open ports 80 or 443. If I restart again it comes good. Or if I svc -d, wait for the log to show a complete shutdown then svc -u everything works correctly.

I haven't tried putting a delay into the run script itslef, but i don't think that would work. I think what I need is to write a script that does an svc -d, waits to see sutdown complete in the log, then does a svc -u.

If it did a -k should the server fail to stop in a reasonable time, all the better.

Of course we could jsut fix the nsd bug...

Collapse
Posted by Bart Teeuwisse on
Mark,

what version of AOLserver do you run? A previous version did have problems releasing the ports when restarting. There is no need to add a delay in the run script. At least not when you restart with 'svc -du'.

/Bart

Collapse
Posted by Mark Aufflick on
My CVS/Tag file tells me I compiled aolserver_v40_r2.

Is svc -du different to svc -t ?

Collapse
Posted by Randy O'Meara on
Mark,

My svc(8) page says:

       -u     Up. If the service is not running,  start  it.  If  the  service
              stops, restart it.
                                                                                
       -d     Down.  If the service is running, send it a TERM signal and then
              a CONT signal. After it stops, do not restart it.
                                                                                
       -t     Terminate. Send the service a TERM signal.
So, it looks like -du sends TERM then CONT while -t just sends TERM. It doesn't really say if -d and -u are handled serially or in the order of appearance. I've never used it this way.

Bart, do you know?

/R

Collapse
Posted by Bart Teeuwisse on
Randy,

svc executes the flags in the order of appearance. Thus 'svc -hdu' rolls the AOLserver log (when not using multilog), stops the server and restarts it again. The rolled log starts with the message that the server is shutting down followed by the restart.

Where as 'svc -dhu' will stutdown the server, send a SIGHUP which will be missed as the server is down and restart the server. In this case the server restarts but the log won't be rolled.

For the sake of completeness, when using multilog, AOLserver runs in foreground mode and won't write to a log itself. Instead the messages to STDOUT are piped to mutilog who writes it to a different log. Rolling a multilog is done w/ 'svc -a'.

I've ran AOLserver in both configurations. I favor multilog as it garantues that the logs won't exceed a set maximum amount of disk space. With a wrapper script in /etc/init.d both configurations can be controled with 'service <service0> {start|stop|restart|roll|stat}'.

/Bart

Collapse
Posted by Mark Aufflick on
DJB tools are so great, but so darn finicky and weakly documented :(

The good thing about DJB tools is they have all the flexibility and terseness of C. The bad thing about DJB tools is that they have all the flexibility and terseness of C.

Collapse
Posted by Mark Aufflick on
Bart - how does your server behave when you click on a restart server link within the ACS admin pages? Mine fails to come up with the symptoms in the log file as discussed above.

I sometimes want to restart the server when I'm behind a proxy on a box with no ssh ability.

Collapse
Posted by Dave Hwang on
Mark, have you checked file ownership of the log files? When I've had this this problem, it's because aolserver can't open the log file. And that's usually due to the fact that started up the test server on the high-port as myself (and not nsadmin) but when daemontools tries to start it as nsadmin, it's not able to open the existing log file with write permissions.
Collapse
Posted by Bart Teeuwisse on
Mark,

I don't restart my server through the Admin pages as I have always ssh access. However, your problem might be a premission issue. As the daemontools documentation states svc runs only with the permissions of the user and its primary group. All auxiliary groups of the user are ignored.

This can have unexpected side effects such as being able to execute the run file from the command line w/o errors but not when using svc. Make sure that root:root has read access to your config.tcl configuration file and the file with the addresses and ports nsd should listen to.

If root:root can't read the address configuration file then nsd won't prebind any addresses and thus fail to listen to ports http and https. There is not error message in the log when nsd can't read the address configuration file. There simply isn't the usual prebind message.

I hope this helps.

/Bart