Forum OpenACS Q&A: Re: My production site is down

Collapse
Posted by Brian Fenton on
Co-incidentally I had to implement error log rolling today. Here's what I did based on an ACS 3 version I had working written by mailto:arjun@openforce.net :

In your server's .tcl file put these 2 lines after ns_section ns/parameters:

ns_param  logroll            on
ns_param  maxbackup          30

Then create 2 files in /acs-tcl/tcl, one called logroll-init.tcl and the other called logroll-procs.tcl.

logroll-procs.tcl contains the following:
#
# logroll.tcl - Rolls the server log on a daily basis
#
# mailto:arjun@openforce.net
# May 17, 2001
# version 0.1
#
# Note: This script is for rolling the _server_ log not the _access_ log!
ad_library {
    TCL library for the rolling error log.

    @creation-date 23/01/2003
}

#
# Directions
# ----------
# 1. Set the MaxBackup ns_param in the "ns/parameters" section of your config
# file to the desired number of backups. NOTE: The AOLServer documentation
# incorrectly calls this param "LogMaxBackup"! You can verify this param is set
# correctly by seeing something like "Notice: conf: [ns/parameters]maxbackup = 6"
# in your server log.
#
#
# Further Work
# ------------
# - Verify the log got rolled, if not send email
# - Check for disk space
# - Allow for non-daily intervals
# - scp logs to a remote site(s)

# You don't have to edit anything below this line
ad_proc -public roll_server_log {
} {
  Proc to roll server error log
}  {
set MaxBackupValue [ns_config -int "ns/parameters" MaxBackup]
if {$MaxBackupValue == ""} {
    # The default is 10
    set MaxBackupValue 10
}

ns_log Notice "logroll.tcl: About to roll server log. ns_param MaxBackup = $MaxBackupValue "
ns_logroll
ns_log Notice "logroll.tcl: Just rolled server log"
}

logroll-init.tcl contains the following:
# Adjust the paramaters below. Hours are between 0-23,
#  and minutes are between 0-59
set hour 16
set minute 00

ns_log Notice "Scheduling logroll"

# Schedule "roll_server_log" to run at the desired time
ns_schedule_daily $hour $minute roll_server_log

Then just bounce your AOLserver.

Collapse
Posted by Jade Rubick on
Brian, thank you. That worked very well.

Shouldn't this be something we do by default?

WHEREAS:

1) unless you implement log rolling, your server will go down

2) hapless, pitiful souls who do not know they have to implement log rolling will fill our esteemed Forums with pleading requests for assistance, and bother those with better things to do with their time.

3) A solution is easy to implement, and easy to control.

SHOULD WE NOT THEREFORE:

1) add it to the distribution?

Collapse
Posted by Joel Aufrecht on
added log-rolling to the to-do list for docs - I can just add it to the default config.tcl file, right?
Collapse
Posted by Tilmann Singer on
As far as I know aolserver can only roll the access logs automatically, not the error log. Which is a pity. I would like to hear if there is any common solution for this, or if aolservers newer than 3.3ad13 can do it.
Collapse
Posted by Bart Teeuwisse on
Tilmann,

one way to roll the access log is mentioned in http://openacs.org/forums/message-view?message_id=34256.

Here's my Tcl version:


#
# logroll.tcl.postload - Rolls the server log on the same basis as the access log.
#
# bart.teeuwisse@thecodemill.biz
# Oct 07, 2001
# version 0.1 based on prior work by
# arjun@openforce.net 
# May 17, 2001
# version 0.1 
#
# Note: This script is for rolling the _server_ log not the _access_ log!
#
# Directions
# ----------
# 1. Set the "ServerLog" in the ns/parameters section and the 
# "RollDay", "RollHour", "RollFmt" parameters in the
# ns/server/'yourserver'/module/nslog section of your config file.
#
# 2. Place this script a Tcl directory sourced at server startup
#
# Further Work
# ------------
# - Verify the log got rolled, if not send email
# - Check for disk space
# - scp logs to a remote site(s)

# Roll the server log and give it an extension of the current date and time.

proc roll_server_log {serverlog rollfmt} {
    ns_log Notice "logroll.tcl: About to roll server log."
    ns_logroll
    set date [clock format [clock seconds] -format $rollfmt]
    if {[file exists "$serverlog.000"]} {
	file rename "$serverlog.000" "$serverlog.$date"
	ns_log Notice "logroll.tcl: Just rolled server log into $serverlog.$date"
    } else {
	ns_log Warning "logroll.tcl: Just rolled server log but couldn't move it to $serverlog.$date"
    }
}

# Create argument list

set args [list]

# Find out where the log is stored.

lappend args [ns_config "ns/parameters" ServerLog]

# Roll the log when the access log is being rolled.

set rollday [ns_config "ns/server/[ns_info server]/module/nslog" RollDay]
set rollhour [ns_config -int "ns/server/[ns_info server]/module/nslog" RollHour]
set rollminute 0

# Use the same roll format as the access log.

lappend args [ns_config "ns/server/[ns_info server]/module/nslog" RollFmt]

if {$rollday == "*"} {

    # Schedule "roll_server_log" to run at the desired time

    ns_schedule_daily $rollhour $rollminute roll_server_log $args
} else {

    # Schedule "roll_server_log" to run only on RollDay days.

    ns_schedule_weekly $rollday $rollhour $rollminute roll_server_log $args
}

Place this file in /web/'yourserver'/tcl directory, follow the directions in the header and restart the server. Et voila!

/Bart

Collapse
Posted by Andrew Piskorski on
Tilmann, please read the earlier posts above in this same thread! AOLserver is quite capable of rolling both the access and the error/server logs itself. (AFAIK this applies to all versions, but definitely applies to AOLserver 3.3.) The methods for the two are gratuitously different, unfortunately, but they're both very easy. And both the easiest and second easiest ways to roll the error log (ns_logroll, SIGHUP) are both explained above, particularly in my Jan. 2003 post.
Collapse
31: default OpenACS solution (response to 25)
Posted by Andrew Piskorski on
added log-rolling to the to-do list for docs - I can just add it to the default config.tcl file, right?

Joel, you're on the right track but it's not quite that simple.

First, here's all the stuff I have in my AOLserver config.tcl related to both the server and access logs. You may want to change some of the settings for the OpenACS default config.tcl (I'm not sure), but you should at least look at what all of these do:

ns_section ns/parameters 
 
  ns_param serverlog $error_log_full_filename($which) 
  # Number of days to keep old error logs around: 
  ns_param maxbackup 7 
 
ns_section ns/server/$server_name/module/nslog 
 
  ns_param File $access_log_full_filename 
  # Number of days to keep old access logs around: 
  ns_param maxbackup 7 
 
  ns_param EnableHostnameLookup Off 
  ns_param LogRefer Off 
  ns_param LogUserAgent Off 
  ns_param maxbackup 7 
  ns_param RollDay * 
  ns_param RollFmt %Y-%m-%d-%H:%M 
  ns_param RollHour 0 
  ns_param RollOnSignal On 
  ns_param RollLog On 
  ns_param ExtendedHeaders X-User-Tracking 

The settings above will take care of rolling the access log without doing anything else. But to roll the server log, you need to schedule ns_logroll (usually once per night at midnight) to roll the log. See my old Jan. 18 2003 definition of my dtk_roll_server_log proc above.

I think using a simple helper proc like that is a bit nicer than scheduling ns_logroll directly, because it writes a notice to the log before and after telling you what's going on. So you just need a proc like that (probably named "ad_roll_server_log"), and the call to ad_schedule_proc.

Hm, I don't see anywhere better to put it, so, ad_roll_server_log should probably just go into packages/acs-tcl/tcl/utilities-proc.tcl, and the call to ad_schedule_proc should go into the matching utilities-init.tcl file.

Once that's all done... then the only thing to put into the docs is pointers to those places, and telling people how to change it in case they need to.