Forum OpenACS Q&A: mod_log_spread for AOLserver?

Posted by Tim The Evil on 05/17/01 06:28 PM

I have a (sort of) stupid question. I'm running mod_log_spread for distributed logging amongst our web, Real, and Quicktime servers, or at least the ones that run Apache on port 80. I'm setting up a pay- as-you-go service for digital video hosting and in the interest of rapid development and scalability, am using OpenACS (duh) on a nice fast dual-Coppermine box with huge mirrored drives. So the question arises, why waste up to 30% of the performance I get just writing AOLserver logs to disk? I have spreadlogd running on my analysis host for just such purposes -- to free up the webservers to read from disk, not write to it. Has anyone built a plugin or module or whatever for AOLserver/OpenNSD to use Spread? If not, what would be involved?

I'll do the damn port, I just need some guidance. Right now it isn't critical but I'll be damned if I'm going to collect logs from multiple hosts with the ol' RSA-exchanged-scp method.

Mod_log_spread: http://www.backhand.org /mod_log_spread/
Spread toolkit: http://www.spread.org/

Thanks!

Tim

2: Response to mod_log_spread for AOLserver? (response to 1)

Posted by Jerry Asher on 05/17/01 07:10 PM

I don't know that anyone has created a spread module, but it may not be too hard. You may be able to create your own, inexpensive spread (sorry), by:

recycling the spread daemon from mod_log_spread to create a spread daemon on your local machine
using Rob Mayoff's dqd_log module to get AOLserver to send access log entries directly to an external program".
creating that external program to forward log messages to the spread daemon (by removing anything but the spread communication code from mod_log_spread.c)

In the long run, once the basic system is working, it would be pretty easy to change ns_log.c to interact directly with the spread daemon.

One question: mod_log_spread looks like it addresses what AOLserver refers to as the access log. What is your strategy regarding the AOLserver server log? (Which could also be spread, but would require mods into a different piece of AOLserver.)

3: Logs for clusters (response to 1)

Posted by David Walker on 05/17/01 07:25 PM

Personally I don't think the server log should be spread.

The access logs need to be examined together for a cluster of servers but the server log applies more to a single machine for troubleshooting purposes.

If one did want to "clusterize" the server log I would think a strategy using something like HP Openview with SNMP or remote syslog would be the way to go.

The dqd_log module sounds interesting. I'll have to look into it some more. I found it at http://www.dqd.com/~mayoff/aolserver/

4: Response to mod_log_spread for AOLserver? (response to 1)

Posted by Tim The Evil on 05/17/01 08:59 PM

Cool, I will try it with dqd_log and if that works maybe I'll hack up a C version as a module in the core distribution later on. I only want to use the access logs for analysis/billing purposes, the error logs are typically viewed on the host when I work on them. (eg. NetSaint or a customer will notify me, then I'll unbind the host from the loadbalancer and fix it; at that point performance is not an issue relative to correctness 😉).

Thanks much for the speedy responses and I'll post my experiences later on when I have some data points to report.