maintenance.xml
Delivered as text/xml
[ hide source ] | [ make this the default ]
File Contents
<?xml version='1.0' ?>
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd" [
<!ENTITY % myvars SYSTEM "../variables.ent">
%myvars;
]>
<chapter id="maintenance-web">
<title>Production Environments</title>
<authorblurb>
<para>by <ulink url="mailto:joel@aufrecht.org">Joel Aufrecht</ulink></para>
</authorblurb>
<sect1 id="install-openacs-keepalive" xreflabel="Keep AOLserver alive">
<title>Starting and Stopping an OpenACS instance.</title>
<para>The simplest way to start and stop and OpenACS site is to run the startup shell script provided, <computeroutput>/var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/etc/daemontools/run</computeroutput>. This runs as a regular task, and logs to the logfile. To stop the site, kill the script.</para>
<para>A more stable way to run OpenACS is with a "keepalive" mechanism of some sort, so that whenever the server halts or is stopped for a reset, it restarts automatically. This is recommended for development and production servers.</para>
<para>The Reference Platform uses Daemontools to control AOLserver. A simpler method, using <computeroutput>init</computeroutput>, is <link linkend="install-openacs-inittab">here</link>.</para>
<orderedlist>
<listitem>
<para>Daemontools must already be installed. If not, <link linkend="install-daemontools">install it</link>.</para>
</listitem>
<listitem>
<para>Each service controlled by daemontools must have a
directory in <computeroutput>/service</computeroutput>. That
directory must have a file called
<computeroutput>run</computeroutput>. It works like this:</para>
<itemizedlist>
<listitem><para>The <computeroutput>init</computeroutput> program starts every
time the computer is booted. </para></listitem>
<listitem><para>A line in <computeroutput>init</computeroutput>'s configuration
file, <computeroutput>/etc/inittab</computeroutput>, tells init to
run, and to restart if necessary,
<computeroutput>svscanboot</computeroutput>.</para></listitem>
<listitem><para><computeroutput>svscanboot</computeroutput> checks
the directory <computeroutput>/service</computeroutput>
every few seconds.</para>
</listitem>
<listitem><para>If it sees a subdirectory there, it
looks for a file in the subdirectory called
<computeroutput>run</computeroutput>. If it finds a run file, it creates a <computeroutput>supervise</computeroutput> process</para>
</listitem>
<listitem>
<para><computeroutput>supervise </computeroutput> executes the run script. Whenever the run script stops, <computeroutput>supervise</computeroutput> executes it again. It also creates additional control files in the same directory. </para>
</listitem>
</itemizedlist>
<para>Hence, the AOLserver
instance for your development server is started by the file
<computeroutput>/service/$OPENACS_SERVICE_NAME/run</computeroutput>.
But we use a symlink to make it easier to add and remove
stuff from the <computeroutput>/service</computeroutput>, so
the actual location is
<computeroutput>/var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>etc/daemontools/run</computeroutput>.</para>
<para>Daemontools creates additional files and directories to track status and
log. A daemontools directory is included in the OpenACS tarball at
<computeroutput>/var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/etc/daemontools</computeroutput>. To use it, first ill any existing AOLserver instances. As root, link the <computeroutput>daemontools</computeroutput> directory into the <computeroutput>/service</computeroutput> directory. Daemontools' <computeroutput>svscan</computeroutput> process checks this directory every five seconds, and will quickly execute <computeroutput>run</computeroutput>.</para>
<screen>[$OPENACS_SERVICE_NAME etc]$ <userinput>killall nsd</userinput>
nsd: no process killed
[$OPENACS_SERVICE_NAME etc]$ <userinput>emacs /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/etc/daemontools/run</userinput>
[$OPENACS_SERVICE_NAME etc]$ <userinput>exit</userinput>
[root root]# <userinput>ln -s /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/etc/daemontools/ /service/<replaceable>$OPENACS_SERVICE_NAME</replaceable></userinput></screen>
<para>Verify that AOLserver is running.</para>
<screen>[root root]#<userinput> ps -auxw | grep nsd</userinput>
<replaceable>$OPENACS_SERVICE_NAME</replaceable> 5562 14.4 6.2 22436 15952 ? S 11:55 0:04 /usr/local/aolserver/bin/nsd -it /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/etc/config.tcl -u serve
root 5582 0.0 0.2 3276 628 pts/0 S 11:55 0:00 grep nsd
[root root]#</screen>
</listitem>
<listitem>
<para>The user <replaceable>$OPENACS_SERVICE_NAME</replaceable> can now control the service <replaceable>$OPENACS_SERVICE_NAME</replaceable> with these commands:</para>
<itemizedlist>
<listitem><para>
<computeroutput>svc -d /service/<replaceable>$OPENACS_SERVICE_NAME</replaceable></computeroutput> -
Bring the server down
</para></listitem>
<listitem><para>
<computeroutput>svc -u /service/<replaceable>$OPENACS_SERVICE_NAME</replaceable></computeroutput> -
Start the server up and leave it in keepalive mode.
</para></listitem>
<listitem><para>
<computeroutput>svc -o /service/<replaceable>$OPENACS_SERVICE_NAME</replaceable></computeroutput> -
Start the server up once. Do not restart it if it stops.
</para></listitem>
<listitem><para>
<computeroutput>svc -t /service/<replaceable>$OPENACS_SERVICE_NAME</replaceable></computeroutput> -
Stop and immediately restart the server.
</para></listitem>
<listitem><para>
<computeroutput>svc -k /service/<replaceable>$OPENACS_SERVICE_NAME</replaceable></computeroutput> -
Sends the server a KILL signal. This is like KILL -9. AOLserver
exits immediately. If svc -t fails to fully kill AOLserver, use
this option. This does not take the server out of keepalive mode, so it should still bounce back up immediately.
</para></listitem>
</itemizedlist>
</listitem>
<listitem>
<para>Install a script to automate the stopping and starting
of AOLserver services via daemontools. You can then restart a service via <computeroutput>restart-aolserver <replaceable>$OPENACS_SERVICE_NAME</replaceable></computeroutput></para>
<screen>[root root]# <userinput>cp /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/packages/acs-core-docs/www/files/restart-aolserver-daemontools.txt /usr/local/bin/restart-aolserver</userinput>
[root root]# <userinput>chmod 755 /usr/local/bin/restart-aolserver</userinput>
[root root]#</screen>
</listitem>
<listitem>
<para>
At this point, these commands will work only for the
<computeroutput>root</computeroutput> user. Grant permission for the <computeroutput>web</computeroutput> group to use <computeroutput>svc</computeroutput> commands on the <emphasis><replaceable>$OPENACS_SERVICE_NAME</replaceable></emphasis> server.</para>
<screen>[root root]# <userinput>/usr/local/bin/svgroup web /service/<replaceable>$OPENACS_SERVICE_NAME</replaceable></userinput>
[root root]#</screen>
</listitem>
<listitem>
<para>Verify that the controls work. You may want to <computeroutput>tail -f /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/log/<replaceable>$OPENACS_SERVICE_NAME</replaceable>-error.log</computeroutput> in another window, so you can see what happens when you type these commands.
</para>
<para>
More information can be found on the <ulink
url="http://panoptic.com/wiki/aolserver/How_to_start_stop_AOLserver_using_Daemontools">AOLserver
Daemontools</ulink> page.
</para>
</listitem>
</orderedlist>
<table frame='all'><title>How it Works</title>
<tgroup cols='6' align='center' colsep='1' rowsep='1'>
<thead>
<row>
<entry>Program</entry>
<entry>Invoked by this program ...</entry>
<entry>... using this file</entry>
<entry>Where to find errors</entry>
<entry>Log goes to</entry>
<entry>Use these commands to control it</entry>
</row>
</thead>
<tbody>
<row>
<entry>svscanboot
</entry>
<entry>init</entry>
<entry>/etc/inittab</entry>
<entry>ps -auxw | grep readproctitle</entry>
<entry>n/a</entry>
<entry></entry>
</row>
<row>
<entry>aolserver</entry>
<entry><computeroutput></computeroutput>supervise
(a child of svscanboot)</entry>
<entry>/service/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/run</entry>
<entry>/var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/log/error.log</entry>
<entry>/var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/log/$OPENACS_SERVICE_NAME.log</entry>
<entry>svc -k /service/<replaceable>$OPENACS_SERVICE_NAME</replaceable></entry>
</row>
<row>
<entry>postgresql</entry>
<entry>Redhat init scripts during boot</entry>
<entry>/etc/init.d/postgresql</entry>
<entry>/usr/local/pgsql/data/server.log</entry>
<entry></entry>
<entry>service postgresql start (Red Hat), /etc/init.d/postgresql start (Debian)</entry>
</row>
</tbody>
</tgroup>
</table>
</sect1>
<sect1 id="install-openacs-inittab" xreflabel="Editing inittab">
<title>AOLserver keepalive with inittab</title>
<para>This is an alternative method for keeping the AOLserver
process running. The recommended method is to <link
linkend="install-openacs-keepalive">run AOLserver
supervised</link>.</para>
<para>
This step should be completed as root. This can break every service
on your machine, so proceed with caution.
</para>
<itemizedlist>
<listitem><para>
There are 2 general steps to getting this working.
</para>
<orderedlist>
<listitem><para>
Install a script called
<computeroutput>restart-aolserver</computeroutput>. This
script doesn't actually restart AOLserver - it just kills
it.
</para></listitem>
<listitem><para>
Ask the OS to restart our service whenever it's not
running. We do this by adding a line to
<computeroutput>/etc/inittab</computeroutput>.
</para></listitem>
</orderedlist>
<para>
Calling <computeroutput>restart-aolserver</computeroutput>
kills our service. The OS notices that our service is not
running, so it automatically restarts it. Thus, calling
<computeroutput>restart-aolserver</computeroutput> effectively
restarts our service.
</para></listitem>
<listitem><para>
Copy this <ulink
url="files/restart-aolserver.txt">file</ulink> into
<computeroutput>/var/tmp/restart-aolserver.txt</computeroutput>.
</para></listitem>
<listitem><para>
This script needs to be SUID-root, which means
that the script will run as root. This is necessary to ensure
that the AOLserver processes are killed regardless of who owns
them. However, the script should be executable by the
<computeroutput>web</computeroutput> group to ensure that the
users updating the web page can use the script, but that
general system users cannot run the script. You also need to
have Perl installed and also a symbolic link to it in
<computeroutput>/usr/local/bin</computeroutput>.
</para>
<programlisting>
[joeuser ~]$ su -
Password: ***********
[root ~]# cp /var/tmp/restart-aolserver.txt /usr/local/bin/restart-aolserver
[root ~]# chown root.web /usr/local/bin/restart-aolserver
[root ~]# chmod 4750 /usr/local/bin/restart-aolserver
[root ~]# ln -s /usr/bin/perl /usr/local/bin/perl
[root ~]# exit</programlisting>
</listitem>
<listitem><para>
Test the <computeroutput>restart-aolserver</computeroutput>
script. We'll first kill all running servers to clean the
slate. Then, we'll start one server and use
<computeroutput>restart-aolserver</computeroutput> to kill
it. If it works, then there should be no more servers
running. You should see the following lines. </para>
<programlisting>
[joeuser ~]$ killall nsd
nsd: no process killed
[joeuser ~]$ /usr/local/aolserver/bin/nsd-postgres -t ~/var/lib/aolserver/<emphasis>birdnotes</emphasis>/nsd.tcl
[joeuser ~]$ restart-aolserver <emphasis>birdnotes</emphasis>
Killing 23727
[joeuser ~]$ killall nsd
nsd: no process killed</programlisting>
<para>
The number 23727 indicates the process id(s) (PIDs) of the
processes being killed. It is important that <emphasis
role="strong">no processes are killed</emphasis> by the second
call to <computeroutput>killall</computeroutput>. If there are
processes being killed, it means that the script is not
working.</para></listitem>
<listitem><para>
Assuming that the <computeroutput>restart-aolserver</computeroutput>
script worked, login as root and open
<computeroutput>/etc/inittab</computeroutput> for
editing. </para>
<programlisting>
[joeuser ~]$ su -
Password: ************
[root ~]# emacs -nw /etc/inittab</programlisting>
</listitem>
<listitem><para>
Copy this line into the bottom of the file as a template,
making sure that the first field
<computeroutput>nss1</computeroutput> is unique.
</para>
<programlisting>
nss1:345:respawn:/usr/local/aolserver/bin/nsd-postgres -i -u nobody -g web -t /home/<emphasis>joeuser</emphasis>/var/lib/aolserver/<emphasis>birdnotes</emphasis>/nsd.tcl</programlisting>
</listitem>
<listitem><para>
<emphasis role="strong">Important:</emphasis> Make sure there is a
newline at the end of the file. If there is not a newline at
the end of the file, the system may suffer catastrophic
failures.
</para></listitem>
<listitem><para>
Still as root, enter the following command to re-initialize
<computeroutput>/etc/inittab</computeroutput>. </para>
<programlisting>
[root ~]# killall nsd
nsd: no process killed
[root ~]# /sbin/init q</programlisting>
</listitem>
<listitem><para>
See if it worked by running the
<computeroutput>restart-aolserver</computeroutput> script
again. </para>
<programlisting>
[root ~]# restart-aolserver <emphasis>birdnotes</emphasis>
Killing 23750</programlisting>
</listitem>
</itemizedlist>
<para>
If processes were killed, congratulations, your server is now
automated for startup and shutdown.
</para>
</sect1>
<sect1 id="install-next-add-server">
<title>Running multiple services on one machine</title>
<formalpara>
<title>Services on different ports</title>
<para>To run a different service on another port but the same
ip, simply repeat <xref linkend="openacs"/> replacing
<replaceable>$OPENACS_SERVICE_NAME</replaceable>, and change the
<programlisting>set httpport 8000
set httpsport 8443 </programlisting>
to different values.</para>
</formalpara>
<formalpara>
<title>Services on different hostnames</title>
<para>For example, suppose you want to support
<computeroutput>http://service0.com</computeroutput> and
<computeroutput>http://bar.com</computeroutput> on the same
machine. The easiest way is to assign each one a different ip
address. Then you can install two services as above, but with
different values for
<programlisting>set hostname [ns_info hostname]
set address 127.0.0.1 </programlisting>
</para>
</formalpara>
<para>If you want to install two services with different hostnames sharing the same ip, you'll need nsvhr to redirect requests
based on the contents of the tcp headers. See <ulink
url="http://borkware.com/rants/aolserver-vhosting/">AOLserver
Virtual Hosting with TCP</ulink> by <ulink url="mailto:markd@borkware.com">markd</ulink>.
</para>
</sect1>
<sect1 id="high-avail">
<title>High Availability/High Performance Configurations</title>
<para>See also <xref linkend="remote-postgres"/>.</para>
<figure>
<title>Multiple-server configuration</title>
<mediaobject>
<imageobject>
<imagedata fileref="images/hpha.png" format="PNG" align="center"/>
</imageobject>
</mediaobject>
</figure>
</sect1>
<sect1 id="maintenance-deploy">
<title>Staged Deployment for Production Networks</title>
<authorblurb>
<para><phrase role="cvstag">($Id: maintenance.xml,v 1.35.2.4 2023/07/10 08:36:09 gustafn Exp $)</phrase></para>
<para>By <ulink url="mailto:joel@aufrecht.org">Joel Aufrecht</ulink></para>
</authorblurb>
<para>This section describes two minimal-risk methods for deploying changes on a production network. The important characteristics of a safe change deployment include: (THIS SECTION IN DEVELOPMENT)</para>
<itemizedlist>
<listitem>
<para>Control: You know for sure that the change you are making is the change that you intend to make and is the change that you tested.</para>
</listitem>
<listitem>
<para>Rollback: If anything goes wrong, you can return to the previous working configuration safely and quickly.</para>
</listitem>
</itemizedlist>
<sect2>
<title>Method 1: Deployment with CVS</title>
<para>With this method, we control the files on a site via
CVS. This example uses one developmental server (service0-dev) and one
production server (service0). Depending on your needs, you can also
have a staging server for extensive testing before you go
live. The only way files should move between the server
instances is via cvs.</para>
<para>To set up a developmental installation, first set up
either your developmental installation or your production
installation, and follow the instructions for committing your
files to CVS. We'll assume in this example that you set up the
production server (service0). To set up the developmental instance,
you then follow the install guide again, this time creating a new
user (service0-dev) that you'll use for the new installation. To get
the files for service0-dev, you check them out from cvs (check out
service0). </para>
<programlisting>
su - service0-dev
co -d /cvsroot service0
mv service0 /var/lib/aolserver/service0-dev
ln -s /home/service0-dev/web /var/lib/aolserver/service0-dev
emacs web/etc/config.tcl
emacs web/etc/daemontools/run
</programlisting>
<para>In the config.tcl file, you'll probably want to pay attention
the rollout support section. That will ensure that email on your
developmental server will not be sent out to the general world.</para>
<para>Also, instead of going through the OpenACS online
installer, you'll actually load live data into your production
server. </para>
<para>You can even automate the process of getting live data
from your production server. Copy something like this to
/home/service0-dev/bin and put it in service0-dev's crontab to
run once a night. You'll need to make sure the database backups
are set up in service0's crontab, and that if the servers are on
different physical machines, that the database backup is copied
to the developmental machine once per night.</para>
<programlisting>
/usr/local/bin/svc -d /service/service0-dev
/bin/sleep 60
# this deletes the dev database!
/usr/local/pgsql/bin/dropdb service0-dev
/usr/local/pgsql/bin/createdb -E UNICODE service0-dev
# this is not necessary from Postgres 7.4 on
/usr/local/pgsql/bin/psql -f /var/lib/aolserver/service0-dev/packages/acs-kernel/sql/postgresql/postgresql.sql service0
mv /var/lib/aolserver/service0/database-backup/service0-nightly-backup.dmp.gz /var/lib/aolserver/service0-dev/database-backup/service0-nightly-backup-old.dmp.gz
/bin/gunzip /var/lib/aolserver/service0-dev/database-backup/service0-nightly-backup.dmp.gz
/usr/bin/perl -pi -e "s/^\\connect service0$/\\connect service0-dev/" /var/lib/aolserver/service0-dev/database-backup/service0-nightly-backup.dmp
/usr/local/pgsql/bin/psql service0-dev < /var/lib/aolserver/service0-dev/database-backup/service0-nightly-backup.dmp
/usr/local/bin/svc -u /service/service0-dev
/bin/gzip /var/lib/aolserver/service0-dev/database-backup/service0-nightly-backup-old.dmp
</programlisting>
<para>Your developmental server will always have data about a day
old.</para>
<para>To make changes on service0-dev:</para>
<programlisting>
1) change the file on service0-dev as desired
2) test the new file
3) commit the file:
if the file is /var/lib/aolserver/service0-dev/www/index.adp, do:
cd /var/lib/aolserver/service0-dev/www
cvs diff index.adp (this is optional; it's just a
reality check)
the lines starting > will be added and the lines
starting < will be removed, when you commit
if that looks okay, commit with:
cvs -m "changing text on front page for February conference" index.adp
the stuff in -m "service0" is a comment visible only from within cvs commands
</programlisting>
<para>To make these changes take place on service0:</para>
<programlisting>
4) update the file on production:
cd /var/lib/aolserver/service0/www
cvs up -Pd index.adp</programlisting>
<para>If you make changes that require changes to the database,
test them out first on service0-dev, using either -create.sql or
upgrade scripts. Once you've tested them, you then update and
run the upgrade scripts from the package manager. </para>
<para>The production site can run "HEAD" from cvs.</para>
<para>The drawback to using HEAD as the live code is that you cannot commit new work on the development server without erasing the definition of 'working production code.' So a better method is to use a tag. This guarantees that, at any time in the future, you can retrieve exactly the same set of code. This is useful for both of the characteristics of safe change deployment. For control, you can use tags to define a body of code, test that code, and then know that what you are deploying is exactly that code. For rollback, you can use return to the last working tag if the new tag (or new, untagged changes) cause problems. .... example of using tags to follow ...</para>
</sect2>
<sect2>
<title>Method 2: A/B Deployment</title>
<para>The approach taken in this section is to always create a new service with the desired changes, running in parallel with the existing site. This guarantees control, at least at the final step of the process: you know what changes you are about to make because you can see them directly. It does not, by itself, guarantee the entire control chain. You need additional measures to make sure that the change you are making is exactly and completely the change you intended to make and tested previously, and nothing more. Those additional measures typically take the form of source control tags and system version numbers. The parallel-server approach also guarantees rollback because the original working service is not touched; it is merely set aside.</para>
<para>This approach can has limitations. If the database or filesystem regularly receiving new data, you must interrupt this function or risk losing data in the shuffle. It also requires extra steps if the database will be affected.</para>
<sect3>
<title>Simple A/B Deployment: Database is not changed</title>
<figure>
<title>Simple A/B Deployment - Step 1</title>
<mediaobject>
<imageobject>
<imagedata fileref="images/simple-deploy-1.png" format="PNG" align="center"/>
</imageobject>
</mediaobject>
</figure>
<figure>
<title>Simple A/B Deployment - Step 2</title>
<mediaobject>
<imageobject>
<imagedata fileref="images/simple-deploy-2.png" format="PNG" align="center"/>
</imageobject>
</mediaobject>
</figure>
<figure>
<title>Simple A/B Deployment - Step 3</title>
<mediaobject>
<imageobject>
<imagedata fileref="images/simple-deploy-3.png" format="PNG" align="center"/>
</imageobject>
</mediaobject>
</figure>
</sect3>
<sect3>
<title>Complex A/B Deployment: Database is changed</title>
<figure>
<title>Complex A/B Deployment - Step 1</title>
<mediaobject>
<imageobject>
<imagedata fileref="images/complex-deploy-1.png" format="PNG" align="center"/>
</imageobject>
</mediaobject>
</figure>
<figure>
<title>Complex A/B Deployment - Step 2</title>
<mediaobject>
<imageobject>
<imagedata fileref="images/complex-deploy-2.png" format="PNG" align="center"/>
</imageobject>
</mediaobject>
</figure>
<figure>
<title>Complex A/B Deployment - Step 3</title>
<mediaobject>
<imageobject>
<imagedata fileref="images/complex-deploy-3.png" format="PNG" align="center"/>
</imageobject>
</mediaobject>
</figure>
</sect3>
</sect2>
</sect1>
<sect1 id="install-ssl">
<title>Installing SSL Support for an OpenACS service</title>
<para>Debian Users: <computeroutput>apt-get install openssl</computeroutput> before proceeding.</para>
<orderedlist>
<listitem>
<para>Make sure nsopenssl.so is <link linkend="install-nsopenssl">installed</link> for AOLserver.</para>
</listitem>
<listitem>
<para>Uncomment this line from <computeroutput>config.tcl</computeroutput>.</para>
<programlisting>#ns_param nsopenssl ${bindir}/nsopenssl.so
</programlisting>
</listitem>
<listitem id="ssl-certificates" xreflabel="Generate ssl certificates">
<para>Prepare a certificate directory for the service.</para>
<screen>[$OPENACS_SERVICE_NAME etc]$ <userinput>mkdir /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/etc/certs</userinput>
[$OPENACS_SERVICE_NAME etc]$ <userinput>chmod 700 /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/etc/certs</userinput>
[$OPENACS_SERVICE_NAME etc]$
<action>mkdir /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/etc/certs
chmod 700 /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/etc/certs</action></screen>
</listitem>
<listitem>
<para>It takes two files to support an SSL connection. The certificate is the public half of the key pair - the server sends the certificate to browser requesting ssl. The key is the private half of the key pair. In addition, the certificate must be signed by Certificate Authority or browsers will protest. Each web browser ships with a built-in list of acceptable Certificate Authorities (CAs) and their keys. Only a site certificate signed by a known and approved CA will work smoothly. Any other certificate will cause browsers to produce some messages or block the site. Unfortunately, getting a site certificate signed by a CA costs money. In this section, we'll generate an unsigned certificate which will work in most browsers, albeit with pop-up messages.</para>
<para>Use an OpenSSL perl script to generate a certificate and key.</para>
<para>
Debian users: use /usr/lib/ssl/misc/CA.pl instead of /usr/share/ssl/CA
</para>
<para>
macOS users: use perl /System/Library/OpenSSL/misc/CA.pl -newcert instead of /usr/share/ssl/CA
</para>
<screen>[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <userinput>cd /var/lib/aolserver/$OPENACS_SERVICE_NAME/etc/certs</userinput>
[$OPENACS_SERVICE_NAME certs]$ <userinput>perl /usr/share/ssl/misc/CA -newcert</userinput>
Using configuration from /usr/share/ssl/openssl.cnf
Generating a 1024 bit RSA private key
...++++++
.......++++++
writing new private key to 'newreq.pem'
Enter PEM pass phrase:</screen>
<para>Enter a pass phrase for the CA certificate. Then, answer the rest of the questions. At the end you should see this:</para>
<screen>Certificate (and private key) is in newreq.pem
[$OPENACS_SERVICE_NAME certs]$</screen>
<para><computeroutput>newreq.pem</computeroutput> contains our certificate and private key. The key is protected by a passphrase, which means that we'll have to enter the pass phrase each time the server starts. This is impractical and unnecessary, so we create an unprotected version of the key. <emphasis>Security implication</emphasis>: if anyone gets access to the file keyfile.pem, they effectively own the key as much as you do. Mitigation: don't use this key/cert combo for anything besides providing ssl for the web site.</para>
<screen>[root misc]# <userinput>openssl rsa -in newreq.pem -out keyfile.pem</userinput>
read RSA key
Enter PEM pass phrase:
writing RSA key
[$OPENACS_SERVICE_NAME certs]$ </screen>
<para>To create the certificate file, we take the combined file, copy it, and strip out the key.</para>
<screen>[$OPENACS_SERVICE_NAME certs]$ <userinput>cp newreq.pem certfile.pem</userinput>
[root misc]# <userinput>emacs certfile.pem</userinput></screen>
<para>Strip out the section that looks like</para>
<programlisting>-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,F3EDE7CA1B404997
S/Sd2MYA0JVmQuIt5bYowXR1KYKDka1d3DUgtoVTiFepIRUrMkZlCli08mWVjE6T
<emphasis>(11 lines omitted)</emphasis>
1MU24SHLgdTfDJprEdxZOnxajnbxL420xNVc5RRXlJA8Xxhx/HBKTw==
-----END RSA PRIVATE KEY-----</programlisting>
</listitem>
<listitem><para>
If you start up using the etc/daemontools/run script, you will
need to edit this script to make sure the ports are bound for
SSL. Details of this are in the run script.
</para>
</listitem>
</orderedlist>
</sect1>
<sect1 id="analog-setup">
<title>Set up Log Analysis Reports</title>
<para>Analog is a program with processes webserver access logs,
performs DNS lookup, and outputs HTML reports. Analog should
<link linkend="analog-install">already be
installed.</link> A modified configuration file is included in
the OpenACS tarball.</para>
<orderedlist>
<listitem>
<screen>[root src]# <userinput>su - $OPENACS_SERVICE_NAME</userinput>
[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <userinput>cd /var/lib/aolserver/$OPENACS_SERVICE_NAME</userinput>
[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <userinput>mkdir www/log</userinput>
[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <userinput>cp -r /usr/share/analog-5.32/images www/log/</userinput>
[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <action>
su - $OPENACS_SERVICE_NAME
cd /var/lib/aolserver/$OPENACS_SERVICE_NAME
cp /var/lib/aolserver/$OPENACS_SERVICE_NAME/packages/acs-core-docs/www/files/analog.cfg.txt etc/analog.cfg
mkdir www/log
cp -r /usr/share/analog-5.32/images www/log/</action></screen>
<para>Edit
<computeroutput>/var/lib/aolserver/$OPENACS_SERVICE_NAME/etc/analog.cfg</computeroutput> and change the variable in <computeroutput>HOSTNAME "[my
organization]"</computeroutput> to reflect your website title. If you
don't want the traffic log to be publicly visible, change
<computeroutput>OUTFILE /var/lib/aolserver/$OPENACS_SERVICE_NAME/www/log/traffic.html</computeroutput> to use a private
directory. You'll also need to edit all instances of service0 to your $OPENACS_SERVICE_NAME.</para>
</listitem>
<listitem>
<para>Run it.</para>
<screen>[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <userinput>/usr/share/analog-5.32/analog -G -g/var/lib/aolserver/$OPENACS_SERVICE_NAME/etc/analog.cfg</userinput>
/usr/share/analog-5.32/analog: analog version 5.32/Unix
/usr/share/analog-5.32/analog: Warning F: Failed to open DNS input file
/home/$OPENACS_SERVICE_NAME/dnscache: ignoring it
(For help on all errors and warnings, see docs/errors.html)
/usr/share/analog-5.32/analog: Warning R: Turning off empty Search Word Report
[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$</screen>
<para>Verify that it works by browsing to <computeroutput>http://yourserver.test:8000/log/traffic.html</computeroutput></para>
</listitem>
<listitem>
<para>Automate this by creating a file in
<computeroutput>/etc/cron.daily</computeroutput>.</para>
<screen>[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <userinput>exit</userinput>
logout
[root root]# <userinput>emacs /etc/cron.daily/analog</userinput></screen>
<para>Put this into the file:</para>
<programlisting>#!/bin/sh
/usr/share/analog-5.32/analog -G -g/var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/etc/analog.cfg</programlisting>
<screen>[root root]# <userinput>chmod 755 /etc/cron.daily/analog</userinput></screen>
<para>Test it by running the script.</para>
<screen>[root root]# <userinput>sh /etc/cron.daily/analog</userinput></screen>
<para>Browse to <computeroutput>http://<replaceable>yourserver.test</replaceable>/log/traffic.html</computeroutput></para>
</listitem>
</orderedlist>
</sect1>
<sect1 id="uptime">
<title>External uptime validation</title>
<para>The <ulink url="http://uptime.openacs.org/uptime/">OpenACS uptime site</ulink> can monitor your site and send you an email whenever your site fails to respond. If you test the url <computeroutput>http://<replaceable>yourserver.test</replaceable>/SYSTEM/dbtest.tcl</computeroutput>, you should get back the string <computeroutput>success</computeroutput>.</para>
</sect1>
<sect1 id="maint-performance">
<title>Diagnosing Performance Problems</title>
<itemizedlist>
<listitem>
<para>Did performance problems happen overnight, or did they sneak up on
you? Any clue what caused the performance problems (e.g. loading 20K
users into .LRN)</para>
</listitem>
<listitem>
<para>Is the filesystem out of space? Is the machine swapping to disk constantly?</para>
</listitem>
<listitem>
<para>Isolating and solving database problems.</para>
<itemizedlist>
<listitem>
<para>Without daily internal maintenance, most databases slowly degrade in performance. For PostgreSQL, see <xref linkend="install-next-nightly-vacuum"/>. For Oracle, use <computeroutput>exec dbms_stats.gather_schema_stats('SCHEMA_NAME')</computeroutput> (<ulink url="http://www.piskorski.com/docs/oracle.html">Andrew Piskorski's Oracle notes</ulink>).</para>
</listitem>
<listitem>
<para>You can track the exact amount of time each database query on a page takes:</para>
<orderedlist>
<listitem>
<para>Go to <ulink url="/acs-admin/install">Main Site : Site-Wide Administration : Install Software</ulink></para>
</listitem>
<listitem>
<para>Click on "Install New Application" in "Install from OpenACS Repository"</para>
</listitem>
<listitem>
<para>Choose "ACS Developer Support"></para>
</listitem>
<listitem>
<para>After install is complete, restart the server.</para>
</listitem>
<listitem>
<para>Browse to Developer Support, which is automatically mounted at <computeroutput><ulink url="/ds">/ds</ulink></computeroutput>.
</para>
</listitem>
<listitem>
<para>Turn on Database statistics</para>
</listitem>
<listitem>
<para>Browse directly to a slow page and click "Request Information" at the bottom of the page.</para>
</listitem>
<listitem>
<para>This should return a list of database queries on the page, including the exact query (so it can be cut-paste into psql or oracle) and the time each query took.</para>
<figure>
<title>Query Analysis example</title>
<mediaobject>
<imageobject>
<imagedata fileref="images/query-duration.png" format="PNG"/>
</imageobject>
</mediaobject>
</figure>
</listitem>
</orderedlist>
</listitem>
<listitem>
<para>Identify a runaway Oracle query: first, use <userinput>ps aux</userinput> or <userinput>top</userinput> to get the UNIX process ID of a runaway Oracle process.</para>
<para>Log in to SQL*Plus as the admin:</para>
<screen>[<replaceable>$OPENACS_SERVICE_NAME</replaceable> ~]$ svrmgrl
Oracle Server Manager Release 3.1.7.0.0 - Production
Copyright (c) 1997, 1999, Oracle Corporation. All Rights Reserved.
Oracle8i Enterprise Edition Release 8.1.7.3.0 - Production
With the Partitioning option
JServer Release 8.1.7.3.0 - Production
SVRMGR> <userinput>connect internal</userinput>
Password:</screen>
<para>See all of the running queries, and match the UNIX PID:</para>
<programlisting>select p.spid -- The UNIX PID
,s.sid ,s.serial#
,p.username as os_user
,s.username ,s.status
,p.terminal ,p.program
from v$session s ,v$process p
where p.addr = s.paddr
order by s.username ,p.spid ,s.sid ,s.serial# ;</programlisting>
<para>See the SQL behind the oracle processes:</para>
<programlisting>select s.username
,s.sid ,s.serial#
,sql.sql_text
from v$session s, v$sqltext sql
where sql.address = s.sql_address
and sql.hash_value = s.sql_hash_value
--and upper(s.username) like 'USERNAME%'
order by s.username ,s.sid ,s.serial# ,sql.piece ;</programlisting>
<para>To kill a troubled process:</para>
<programlisting>alter system kill session 'SID,SERIAL#'; --substitute values for SID and SERIAL#</programlisting>
<para>(See <ulink url="http://www.piskorski.com/docs/oracle.html">Andrew Piskorski's Oracle notes</ulink>)</para>
</listitem>
<listitem>
<para>Identify a runaway Postgres query. First, logging must be enabled in the database. This imposes a performance penalty and should not be done in normal operation.</para>
<para>Edit the file <computeroutput>postgresql.conf</computeroutput> - its location depends on the PostgreSQL installation - and change</para>
<programlisting>#stats_command_string = false</programlisting>
<para>to</para>
<programlisting>stats_command_string = true</programlisting>
<para>Next, connect to postgres (<computeroutput>psql <replaceable>service0</replaceable></computeroutput>) and <computeroutput>select * from pg_stat_activity;</computeroutput>. Typical output should look like:</para>
<programlisting>
datid | datname | procpid | usesysid | usename | current_query
----------+-------------+---------+----------+---------+-----------------
64344418 | openacs.org | 14122 | 101 | nsadmin | <IDLE>
64344418 | openacs.org | 14123 | 101 | nsadmin |
delete
from acs_mail_lite_queue
where message_id = '2478608';
64344418 | openacs.org | 14124 | 101 | nsadmin | <IDLE>
64344418 | openacs.org | 14137 | 101 | nsadmin | <IDLE>
64344418 | openacs.org | 14139 | 101 | nsadmin | <IDLE>
64344418 | openacs.org | 14309 | 101 | nsadmin | <IDLE>
64344418 | openacs.org | 14311 | 101 | nsadmin | <IDLE>
64344418 | openacs.org | 14549 | 101 | nsadmin | <IDLE>
(8 rows)
openacs.org=></programlisting>
</listitem>
</itemizedlist>
</listitem>
</itemizedlist>
<sect2 id="install-oracle-monitoring">
<title>Creating an appropriate tuning and monitoring environment</title>
<para>
The first task is to create an appropriate environment for finding out
what is going on inside Oracle. Oracle provides Statspack, a package to
monitor and save the state of the v$ performance views. These reports
help finding severe problems by exposing summary data about the Oracle
wait interface, executed queries. You'll find the installation
instructions in $ORACLE_HOME/rdbms/admin/spdoc.txt. Follow the
instructions carefully and take periodic snapshots, this way you'll be
able to look at historical performance data.
</para>
<para>
Also turn on the timed_statistics in your init.ora file, so that
Statspack reports (and all other Oracle reports) are timed, which makes
them a lot more meaningful. The overhead of timing data is about 1% per
Oracle Support information.
</para>
<para>
To be able to get a overview of how Oracle executes a particular query,
install "autotrace". I usually follow the instructions here <ulink
url="http://asktom.oracle.com/~tkyte/article1/autotrace.html">http://asktom.oracle.com/~tkyte/article1/autotrace.html</ulink>.
</para>
<sect3>
<title>Make sure that the Oracle CBO works with adequate statistics</title>
<para>
The Oracle Cost Based optimizer is a piece of software that tries to find
the "optimal" execution plan for a given SQL statement. For that it
estimates the costs of running a SQL query in a particular way (by default
up to 80.000 permutations are being tested in a Oracle 8i). To get an
adequate cost estimate, the CBO needs to have adequate statistics. For
that Oracle supplies the <ulink
url="http://docs.oracle.com/cd/B19306_01/appdev.102/b14258/d_stats.htm#CIHBIEII">dbms_stats
package</ulink>.
</para>
</sect3>
</sect2>
</sect1>
</chapter>
