Weblog Page

Install Oracle

Created by OpenACS community, last modified by Gustaf Neumann 26 Jun 2022, at 04:43 PM

Installing Oracle is optional, if you are installing PostgreSQL.

If you are using new versions of the oacs-5-10 branch of OpenACS 5.10.1, read oacs-5-10-on-oracle-19c for the installation of Oracle 19c and the improved support in the oacs-core packages. The description below is for earlier versions.

An excellent guide for installing Oracle 10g on Linux can be found at www.puschitz.com.

OpenACS 5.2.x will install with Oracle 9i but has not been extensively tested so may still have bugs or tuning issues.

This installation guide attempts to present all of the information necessary to complete an OpenACS installation. We try hard to make all of the steps possible in one pass, rather than having a step which amounts to "go away and develop a profound understanding of software X and then come back and, in 99% of all cases, type these two lines." The exception to our rule is Oracle production systems. This page describes a set of steps to get a working Oracle development server, but it is unsuitable for production systems. If you will be using OpenACS on Oracle in a production environment, you will experience many problems unless you develop a basic understanding of Oracle which is outside the scope of this document.

This document assumes that you are installing Oracle on the same box as AOLserver. For more details on a remote Oracle installation, see Daryl Biberdorf's document.

Defaults

We use the following defaults while installing Oracle.

Get Oracle

You can register and download Oracle (for free) from Oracle Downloads. You need this if you want to use an Oracle database.

Production Oracle systems should run on certified platforms. Follow the metalink note 223718.1to find certified platforms. If you don't have metalink access, take a look at the Oracle on Linux FAQ: http://www.oracle.com/us/technologies/027617.pdf. In summary, free and inexpensive Linux distributions are not certified.

If you don't have an account at OTN get one: you can download the Oracle software from the Oracle Downloads page.

Each Oracle release comes with extensive and usually quite well-written documentation. Your first step should be to thoroughly read the release notes for your operating system and your Oracle version. Find the docs here:

• http://www.oracle.com/technetwork/documentation/oracle8i-085806.html

• http://www.oracle.com/pls/db92/homepage

• https://docs.oracle.com/cd/B19306_01/server.102/b14214/chapter2.htm#g62359

It is generally useful to run a particular Oracle version with its latest patch-set. At the time of writing these were 8.1.7.4 and 9.2.0.5, both of which are considered to be very stable.

To be able to download a patch-set, you need an account on Metalink (not free). You may find the appropriate patch-set by following Andrew's suggestion.

Things to Keep in Mind

Oracle is very well-documented software, the online documentation comes with printable PDFs and full-text search. Altogether there is more than 20.000 pages of documentation, so do not expect to understand Oracle within in a few hours. The best starting pointing into Oracle is the Concepts book. Here's the 8i version and the 9.2 version.

To give you an idea of how configurable Oracle is and how much thought you may need to put into buying the proper hardware and creating a sane setup, you should thoroughly read Cary Millsap's Configuring Oracle Server for VLDB and the Optimal Flexible Architecture standard.

Throughout these instructions, we will refer to a number of configurable settings and advise certain defaults. With the exception of passwords, we advise you to follow these defaults unless you know what you are doing. Subsequent documents will expect that you used the defaults, so a change made here will necessitate further changes later. For a guide to the defaults, please see the section called “Defaults”.

In order for OpenACS to work properly you need to set the environment appropriately.

export ORACLE_BASE=/ora8/m01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/8.1.7
export PATH=$PATH:$ORACLE_HOME/bin
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export ORACLE_SID=ora8
export ORACLE_TERM=vt100
export ORA_NLS33=$ORACLE_HOME/ocommon/nls/admin/data

umask 022

open_cursors = 500

nls_date_format = "YYYY-MM-DD"

For additional resources/documentation, please see this thread and Andrew Piskorski's mini-guide.

Pre-Installation Tasks

Though Oracle 8.1.7 has an automated installer, we still need to perform several manual, administrative tasks before we can launch it. You must perform all of these steps as the root user. We recommend entering the X window system as a normal user and then doing a su -. This command gives you full root access.

• Login as a non-root user and start X by typing startx

  [joeuser ~]$ startx
  

• Open a terminal window type and login as root

  [joeuser ~]$ su -
  Password: ***********
  [root ~]#
  

• Create and setup the oracle group and oracle account

  We need to create a user oracle, which is used to install the product, as well as starting and stopping the database.

  [root ~]# groupadd dba
  [root ~]# groupadd oinstall
  [root ~]# groupadd oracle
  [root ~]# useradd -g dba -G oinstall,oracle -m oracle
  [root ~]# passwd oracle
  

  You will be prompted for the New Password and Confirmation of that password.

• Setup the installation location for Oracle. While Oracle can reside in a variety of places in the file system, OpenACS has adopted /ora8 as the base directory.

  Note: the Oracle install needs about 1 GB free on /ora8 to install successfully.

  [root ~]# mkdir /ora8
  root:/ora8# cd /ora8
  root:/ora8# mkdir -p m01 m02 m03/oradata/ora8
  root:/ora8# chown -R oracle.dba /ora8
  root:/ora8# exit
  

• Set up the oracle user's environment

  • Log in as the user oracle by typing the following:

    [joeuser ~]$ su - oracle
    Password: ********
    

  • Use a text editor to edit the .bash_profile file in the oracle account home directory.

    [oracle ~]$ emacs .bash_profile
    

    You may get this error trying to start emacs:

    Xlib: connection to ":0.0" refused by server
    Xlib: Client is not authorized to connect to Server
    emacs: Cannot connect to X server :0.
    Check the DISPLAY environment variable or use `-d'.
    Also use the `xhost' program to verify that it is set to permit
    connections from your machine.
    

    If so, open a new terminal window and do the following:

    [joeuser ~]$ xhost +localhost
    

    Now, back in the oracle terminal:

    [oracle ~]$ export DISPLAY=localhost:0.0
    [oracle ~]$ emacs .bash_profile
    

    Try this procedure anytime you get an Xlib connection refused error.

  • Add the following lines (substituting your Oracle version number as needed) to .bash_profile:

    export ORACLE_BASE=/ora8/m01/app/oracle
    export ORACLE_HOME=$ORACLE_BASE/product/8.1.7
    export PATH=$PATH:$ORACLE_HOME/bin
    export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
    export ORACLE_SID=ora8
    export ORACLE_TERM=vt100
    export ORA_NLS33=$ORACLE_HOME/ocommon/nls/admin/data
    
    umask 022
    

    Save the file by typing CTRL-X CTRL-S and then exit by typing CTRL-X CTRL-C. Alternatively, use the menus.

  Make sure that you do not add any lines like the following

  # NLS_LANG=american
  # export NLS_LANG
  

  These lines will change the Oracle date settings and will break OpenACS since OpenACS depends on the ANSI date format, YYYY-MM-DD dates.

• Log out as oracle

  [oracle ~]$ exit
  

• Log back in as oracle and double check that your environment variables are as intended. The env command lists all of the variables that are set in your environment, and grep shows you just the lines you want (those with ORA in it).

  [joeuser ~]$ su - oracle
  [oracle ~]$ env | grep ORA
  

  If it worked, you should see:

  ORACLE_SID=ora8
  ORACLE_BASE=/ora8/m01/app/oracle
  ORACLE_TERM=vt100
  ORACLE_HOME=/ora8/m01/app/oracle/product/8.1.7
  ORA_NLS33=/ora8/m01/app/oracle/product/8.1.7/ocommon/nls/admin/data
  

  If not, try adding the files to ~/.bashrc instead of .bash_profile. Then logout and log back in again. Also, be certain you are doing su - oracle and not just su oracle. The - means that .bashrc and .bash_profile will be evaluated.

  Make sure that /bin, /usr/bin, and /usr/local/bin are in your path by typing:

  [oracle ~]$ echo $PATH
  /bin:/usr/bin:/usr/local/bin:/usr/bin/X11:/usr/X11R6/bin:/home/oracle/bin:/ora8/m01/app/oracle/product/8.1.7/bin
  

  If they are not, then add them to the .bash_profile by changing the PATH statement above to PATH=$PATH:/usr/local/bin:$ORACLE_HOME/bin

Installing Oracle 8.1.7 Server

• Log in as oracle and start X if not already running. Start a new terminal:

  [joeuser ~]$ xhost +localhost
  [joeuser ~]$ su - oracle
  Password: **********
  [oracle ~]$ export DISPLAY=localhost:0.0
  

• Find the runInstaller script

  • If you are installing Oracle from a CD-ROM, it is located in the install/linux path from the cd-rom mount point

    [oracle ~]$ su - root
    [root ~]# mount -t iso9660 /dev/cdrom /mnt/cdrom
    [root ~]# exit
    [oracle ~]$ cd /mnt/cdrom
    

  • If you are installing from the tarball, the install script is located in the Oracle8iR2 directory that was created when you expanded the archive.

    [oracle ~]$ cd /where/oracle/Disk1
    

  Check to make sure the file is there.

  oracle:/where/oracle/Disk1$ ls
  doc  index.htm  install  runInstaller  stage  starterdb
  

  If you don't see runInstaller, you are in the wrong directory.

• Run the installer

  oracle:/where/oracle/Disk1$ ./runInstaller
  

  A window will open that welcomes you to the 'Oracle Universal Installer' (OUI). Click on "Next"

  Note

  Some people have had trouble with this step on RedHat 7.3 and 8.0. If so, try the following steps before calling ./runInstaller:

  1. Execute the following command: /usr/i386-glibc21-linux/bin/i386-glibc21-linux-env.sh

  2. Type export LD_ASSUME_KERNEL=2.2.5

• The "File Locations" screen in the OUI:

  • "Source" path should have been prefilled with "(wherever you mounted the CDROM)/stage/products.jar"

  • "destination" path says "/ora8/m01/app/oracle/product/8.1.7"

    If the destination is not correct it is because your environment variables are not set properly. Make sure you logged on as oracle using su - oracle. If so, edit the ~/.bash_profile as you did in the section called “Pre-Installation Tasks”

  • Click "Next" (a pop up window will display Loading Product information).

• The "Unix Group Name" screen in the OUI:

  • The Unix Group name needs to be set to 'oinstall' ( we made this Unix group earlier ).

  • Click "Next"

  • A popup window appears instantly, requesting you to run a script as root:

  • • Debian users need to link /bin/awk to /usr/bin/awk before running the script below

      [joueser ~]$ su -
      [root ~]# ln -s /usr/bin/awk /bin/awk
      

  • Open a new terminal window, then type:

    [joeuser ~]$ su -
    [root ~]# cd /ora8/m01/app/oracle/product/8.1.7
    [root ~]# ./orainstRoot.sh  
    ; You should see:
    Creating Oracle Inventory pointer file (/etc/oraInst.loc)
    Changing groupname of /ora8/m01/app/oracle/oraInventory to oinstall.
    [root ~]# mkdir -p /usr/local/java
    [root ~]# exit
    [joeuser ~]$ exit
    

  • Click "Retry"

• The "Available Products" screen in the OUI:

  • Select "Oracle 8i Enterprise Edition 8.1.7.1.0"

  • Click "Next"

• The "Installation Types" screen

  • Select the "Custom" installation type.

  • Click "Next"

• The "Available Product Components" screen

  • In addition to the defaults, make sure that "Oracle SQLJ 8.1.7.0," "Oracle Protocol Support 8.1.7.0.0," and "Linux Documentation 8.1.7.0.0" are also checked.

  • Click "Next"

  • A progress bar will appear for about 1 minute.

• The "Component Locations" screen in the OUI

  • Click on the "Java Runtime Environment 1.1.8" It should have the path "/ora8/m01/app/oracle/jre/1.1.8"

  • Click "Next"

  • A progress bar will appear for about 1 minute.

• The "Privileged Operation System Groups" screen in the OUI

  • Enter "dba" for "Database Administrator (OSDBA) Group"

  • Enter "dba" for the "Database Operator (OSOPER) Group"

  • Click "Next"

  • A progress bar will appear for about 1 minute.

• The "Authentication Methods" screen

  • Click "Next"

• The next screen is "Choose JDK home directory"

  • Keep the default path: /usr/local/java

  • Click "Next"

• The "Create a Database" screen in the OUI

  • Select "No" as we will do this later, after some important configuration changes.

  • Click "Next"

• The next screen is "Oracle Product Support"

  • TCP should be checked with "Status" listed as Required

  • Click "Next"

• The "Summary" screen in the OUI

  • Check the "Space Requirements" section to verify you have enough disk space for the install.

  • Check that "(144 products)" is in the "New Installations" section title.

  • Click "Install"

  • A progress bar will appear for about 20 - 30 minutes. Now is a good time to take a break.

  • A "Setup Privileges" window will popup towards the end of the installation asking you to run a script as root

  • Run the script. Switch to the oracle user first to set the environment appropriately and then do su to get root privileges, while keeping the oracle user's environment.

    [joeuser ~]$ su - oracle
    Password: *********
    [oracle ~]$ su
    Password: *********
    [root ~]# /ora8/m01/app/oracle/product/8.1.7/root.sh
    ; You should see the following.   
    
    Creating Oracle Inventory pointer file (/etc/oraInst.loc)
    Changing groupname of /ora8/m01/app/oracle/oraInventory to oinstall.
    # /ora8/m01/app/oracle/product/8.1.7/root.sh
    Running Oracle8 root.sh script...
    The following environment variables are set as:
        ORACLE_OWNER= oracle
        ORACLE_HOME=  /ora8/m01/app/oracle/product/8.1.7
        ORACLE_SID=   ora8
    
    Enter the full pathname of the local bin directory: [/usr/local/bin]: 
    
    Press ENTER here to accept default of /usr/local/bin
          
    
    Creating /etc/oratab file...
    Entry will be added to the /etc/oratab file by
    Database Configuration Assistants when a database is created
    Finished running generic part of root.sh script.
    Now product-specific root actions will be performed.
    IMPORTANT NOTE: Please delete any log and trace files previously
                    created by the Oracle Enterprise Manager Intelligent
                    Agent. These files may be found in the directories
                    you use for storing other Net8 log and trace files.
                    If such files exist, the OEM IA may not restart.
    

  • Do not follow the instructions on deleting trace and log files, it is not necessary.

  [root ~]# exit
  [joeuser ~]$ exit
  

• Go back to the pop-up window and click "OK"

• The "Configuration Tools" screen in the OUI

  • This window displays the config tools that will automatically be launched.

• The "Welcome" screen in the "net 8 Configuration Assistant"

  • Make sure the "Perform Typical installation" is not selected.

  • Click "Next"

  • The "Directory Service Access" screen in the "Net 8 Configuration Assistant"

  • Select "No"

  • Click "Next"

• The "Listener Configuration, Listener Name" screen in the "Net 8 Configuration Assistant"

  • Accept the default listener name of "LISTENER"

  • Click "Next"

• The "Listener Configuration, Select Protocols" screen in the "Net 8 Configuration Assistant"

  • The only choice in "Select protocols:" should be "TCP/IP"

  • Click "Next"

• The "Listener Configuration TCP/IP Protocol" screen in the "Net 8 Configuration Assistant"

  • Default Port should be 1521 and selected.

  • Click "Next"

• The "Listener Configuration, More Listeners" screen in the "Net 8 Configuration Assistant"

  • Select "No"

  • Click "Next"

• The "Listener Configuration Done" screen in the "Net 8 Configuration Assistant"

  • Click "Next"

• The "Naming Methods Configuration" screen in the "Net 8 Configuration Assistant"

  • Select "No"

  • Click "Next"

• The "Done" screen in the "Net 8 Configuration Assistant"

  • Click "Finish"

• The "End of Installation" screen in the OUI

  • Click "Exit"

  • Click "Yes" on the confirmation pop up window.

  • The Oracle Universal Installer window should have disappeared!

Congratulations, you have just installed Oracle 8.1.7 Server! However, you still need to create a database which can take about an hour of non-interactive time, so don't quit yet.

Creating the First Database

This step will take you through the steps of creating a customized database. Be warned that this process takes about an hour on a Pentium II with 128 MB of RAM.

Note

RedHat 7.3 and 8.0 users: Before running dbassist, do the following.

1. Download the glibc patch from Oracle Technet into /var/tmp.

2. cd $ORACLE_HOME

3. tar xzf /var/tmp/glibc2.1.3-stubs.tgz

4. ./setup_stubs

• Make sure you are running X. Open up a terminal and su to oracle and then run the dbassist program.

  [joeuser ~]$ xhost +localhost
  [joeuser ~]$ su - oracle
  Password: *********
  [oracle ~]$ export DISPLAY=localhost:0.0
  [oracle ~]$ dbassist
  

• The "Welcome" screen in the Oracle Database Configuration Agent (ODCA)

  • Select "Create a database"

  • Click "Next"

• The "Select database type" screen in the ODCA

  • Select "Custom"

  • Click "Next"

• The "Primary Database Type" window in ODCA

  • Select "Multipurpose"

  • Click "Next"

• The "concurrent users" screen of the ODCA

  • Select "60" concurrent users.

  • Click "Next"

• Select "Dedicated Server Mode", click "Next"

• Accept all of the options, and click Next Oracle Visual Information Retrieval may be grayed out. If so, you can ignore it; just make sure that everything else is checked.

• For "Global Database Name", enter "ora8"; for "SID", also enter "ora8" (it should do this automatically). Click "Change Character Set and select UTF8. Click "Next".

• Accept the defaults for the next screen (control file location). Click "Next"

• Go to the "temporary" and "rollback" tabs, and change the Size (upper-right text box) to 150MB. Click "Next"

• Increase the redo log sizes to 10000K each. Click "Next"

• Use the default checkpoint interval & timeout. Click "Next"

• Increase "Processes" to 100; "Block Size" to 4096 (better for small Linux boxes; use 8192 for a big Solaris machine).

• Accept the defaults for the Trace File Directory. Click "Next"

• Finally, select "Save information to a shell script" and click "Finish" (We're going to examine the contents of this file before creating our database.)

• Click the "Save" button. Oracle will automatically save it to the correct directory and with the correct file name. This will likely be /ora8/m01/app/oracle/product/8.1.7/assistants/dbca/jlib/sqlora8.sh

• It will alert you that the script has been saved successfully.

• Now we need to customize the database configuration a bit. While still logged on as oracle, edit the database initialization script (run when the db loads). The scripts are kept in $ORACLE_HOME/dbs and the name of the script is usually initSID.ora where SID is the SID of your database. Assuming your $ORACLE_HOME matches our default of /ora8/m01/app/oracle/product/8.1.7, the following will open the file for editing.

  [oracle ~]$ emacs /ora8/m01/app/oracle/product/8.1.7/dbs/initora8.ora
  

• Add the following line to the end:

  nls_date_format = "YYYY-MM-DD"
  

• Now find the open_cursors line in the file. If you're using emacs scroll up to the top of the buffer and do CTRL-S and type open_cursors to find the line. The default is 100. Change it to 500.

  open_cursors = 500
  

• Save the file. In emacs, do CTRL-X CTRL-S to save followed by CTRL-X CTRL-C to exit or use the menu.

• At this point, you are ready to initiate database creation. We recommend shutting down X to free up some RAM unless you have 256 MB of RAM or more. You can do this quickly by doing a CRTL-ALT-BACKSPACE, but make sure you have saved any files you were editing. You should now be returned to a text shell prompt. If you get sent to a graphical login screen instead, switch to a virtual console by doing CRTL-ALT-F1. Then login as oracle.

• Change to the directory where the database creation script is and run it:

  [oracle ~]$ cd /ora8/m01/app/oracle/product/8.1.7/assistants/dbca/jlib
  oracle:/ora8/m01/app/oracle/product/8.1.7/assistants/dbca/jlib$ ./sqlora8.sh
  

  In some instances, Oracle will save the file to /ora8/m01/app/oracle/product/8.1.7/assistants/dbca Try running the script there if your first attempt does not succeed.

• Your database will now be built. It will take > 1 hour - no fooling. You will see lots of errors scroll by (like: "ORA-01432: public synonym to be dropped does not exist") Fear not, this is normal.

  Eventually, you'll be returned to your shell prompt. In the meantime, relax, you've earned it.

Acceptance Test

For this step, open up a terminal and su to oracle as usual. You should be running X and Netscape (or other web browser) for this phase.

• You need to download the "Oracle Acceptance Test" file. It's available here and at http://philip.greenspun.com/wtr/oracle/acceptance-sql.txt. Save the file to /var/tmp

• In the oracle shell, copy the file.

  [oracle ~]$ cp /var/tmp/acceptance-sql.txt /var/tmp/acceptance.sql
  

• Once you've got the acceptance test file all set, stay in your term and type the following:

  [oracle ~]$ sqlplus system/manager
  

  SQL*Plus should startup. If you get an ORA-01034: Oracle not Available error, it is because your Oracle instance is not running. You can manually start it as the oracle user.

  [oracle ~]$ svrmgrl
  SVRMGR> connect internal
  SVRMGR> startup
  

• Now that you're into SQL*Plus, change the default passwords for system, sys, and ctxsys to "alexisahunk" (or to something you'll remember):

  SQL> alter user system identified by alexisahunk;
  SQL> alter user sys identified by alexisahunk;
  SQL> alter user ctxsys identified by alexisahunk;
  

• Verify that your date settings are correct.

  SQL> select sysdate from dual;
  

  If you don't see a date that fits the format YYYY-MM-DD, please read the section called “Troubleshooting Oracle Dates”.

• At this point we are going to hammer your database with an intense acceptance test. This usually takes around 30 minutes.

  SQL> @ /var/tmp/acceptance.sql
  
  ; A bunch of lines will scroll by.  You'll know if the test worked if
  ; you see this at the end:
  
  SYSDATE
  ----------
  2000-06-10
  
  SQL>
  

  Many people encounter an error regarding maximum key length:

  ERROR at line 1:
  ORA-01450: maximum key length (758) exceeded
  

  This error occurs if your database block size is wrong and is usually suffered by people trying to load OpenACS into a pre-existing database. Unfortunately, the only solution is to create a new database with a block size of at least 4096. For instructions on how to do this, see the section called “Creating the First Database” above. You can set the parameter using the dbassist program or by setting the DB_BLOCK_SIZE parameter in your database's creation script.

  If there were no errors, then consider yourself fortunate. Your Oracle installation is working.

Automating Startup & Shutdown

You will want to automate the database startup and shutdown process. It's probably best to have Oracle spring to life when you boot up your machine.

• Oracle includes a script called dbstart that can be used to automatically start the database. Unfortunately, the script shipped in the Linux distribution does not work out of the box. The fix is simple. Follow these directions to apply it. First, save dbstart to /var/tmp. Then, as oracle, do the following:

  [oracle ~]$ cp /var/tmp/dbstart.txt /ora8/m01/app/oracle/product/8.1.7/bin/dbstart 
  [oracle ~]$ chmod 755 /ora8/m01/app/oracle/product/8.1.7/bin/dbstart
  

• While you're logged in as oracle, you should configure the oratab file to load your database at start. Edit the file /etc/oratab:

  • You will see this line.

    ora8:/ora8/m01/app/oracle/product/8.1.7:N
    

    By the way, if you changed the service name or have multiple databases, the format of this file is:

    service_name:$ORACLE_HOME:Y || N (for autoload)

  • Change the last letter from "N" to "Y". This tells Oracle that you want the database to start when the machine boots. It should look like this.

    ora8:/ora8/m01/app/oracle/product/8.1.7:Y
    

  • Save the file & quit the terminal.

• You need a script to automate startup and shutdown. Save oracle8i.txt in /var/tmp. Then login as root and install the script. (Debian users: substitute /etc/init.d for /etc/rc.d/init.d throughout this section)

  [oracle ~]$ su -
  [root ~]# cp /var/tmp/oracle8i.txt /etc/rc.d/init.d/oracle8i
  [root ~]# chown root.root /etc/rc.d/init.d/oracle8i
  [root ~]# chmod 755 /etc/rc.d/init.d/oracle8i
  

• Test the script by typing the following commands and checking the output. (Debian Users: as root, do mkdir /var/lock/subsys first)

  [root ~]# /etc/rc.d/init.d/oracle8i stop
  Oracle 8i auto start/stop
  Shutting Oracle8i:
  Oracle Server Manager Release 3.1.7.0.0 - Production
  
  Copyright (c) 1997, 1999, Oracle Corporation.  All
  Rights Reserved.
  
  Oracle8i Enterprise Edition Release 8.1.7.0.1 -
  Production
  With the Partitioning option
  JServer Release 8.1.7.0.1 - Production
  
  SVRMGR> Connected.
  SVRMGR> Database closed.
  Database dismounted.
  ORACLE instance shut down.
  SVRMGR>
  Server Manager complete.
  Database "ora8" shut down.
        
  [root ~]# /etc/rc.d/init.d/oracle8i start
  Oracle 8i auto start/stop
  Starting Oracle8i: 
  SQL*Plus: Release 8.1.7.0.0 - Production on Wed Mar 6 17:56:02 2002
  
  (c) Copyright 2000 Oracle Corporation.  All rights reserved.
  
  SQL> Connected to an idle instance.
  SQL> ORACLE instance started.
  
  Total System Global Area   84713632 bytes
  Fixed Size                    73888 bytes
  Variable Size              76079104 bytes
  Database Buffers            8388608 bytes
  Redo Buffers                 172032 bytes
  Database mounted.
  Database opened.
  SQL> Disconnected
  
  Database "ora8" warm started.
  
  Database "ora8" warm started.
  

• If it worked, then run these commands to make the startup and shutdown automatic.

  • Red Hat users:

    [root ~]# cd /etc/rc.d/init.d/                      
    [root ~]# chkconfig --add oracle8i
    [root ~]# chkconfig --list oracle8i
    ; You should see:
    oracle8i        0:off   1:off   2:off   3:on    4:on    5:on    6:off
    

  • Debian users:

    [root ~]# update-rc.d oracle8i defaults
     Adding system startup for /etc/init.d/oracle8i ...
       /etc/rc0.d/K20oracle8i -> ../init.d/oracle8i
       /etc/rc1.d/K20oracle8i -> ../init.d/oracle8i
       /etc/rc6.d/K20oracle8i -> ../init.d/oracle8i
       /etc/rc2.d/S20oracle8i -> ../init.d/oracle8i
       /etc/rc3.d/S20oracle8i -> ../init.d/oracle8i
       /etc/rc4.d/S20oracle8i -> ../init.d/oracle8i
       /etc/rc5.d/S20oracle8i -> ../init.d/oracle8i
    

  • SuSE users:

    [root ~]# cd /etc/rc.d/init.d
    root:/etc/rc.d/init.d# ln -s /etc/rc.d/init.d/oracle8i K20oracle8i
    root:/etc/rc.d/init.d# ln -s /etc/rc.d/init.d/oracle8i S20oracle8i
    root:/etc/rc.d/init.d# cp K20oracle8i rc0.d
    root:/etc/rc.d/init.d# cp S20oracle8i rc0.d
    root:/etc/rc.d/init.d# cp K20oracle8i rc1.d
    root:/etc/rc.d/init.d# cp S20oracle8i rc1.d 
    root:/etc/rc.d/init.d# cp K20oracle8i rc6.d
    root:/etc/rc.d/init.d# cp S20oracle8i rc6.d
    root:/etc/rc.d/init.d# cp K20oracle8i rc2.d
    root:/etc/rc.d/init.d# cp S20oracle8i rc2.d
    root:/etc/rc.d/init.d# cp K20oracle8i rc3.d
    root:/etc/rc.d/init.d# cp S20oracle8i rc3.d 
    root:/etc/rc.d/init.d# cp K20oracle8i rc4.d  
    root:/etc/rc.d/init.d# cp S20oracle8i rc4.d  
    root:/etc/rc.d/init.d# cp K20oracle8i rc5.d
    root:/etc/rc.d/init.d# cp S20oracle8i rc5.d
    root:/etc/rc.d/init.d# rm K20oracle8i
    root:/etc/rc.d/init.d# rm S20oracle8i
    root:/etc/rc.d/init.d# cd
    [root ~]# SuSEconfig
    Started the SuSE-Configuration Tool.
    Running in full featured mode.
    Reading /etc/rc.config and updating the system...
    Executing /sbin/conf.d/SuSEconfig.gdm...   
    Executing /sbin/conf.d/SuSEconfig.gnprint...
    Executing /sbin/conf.d/SuSEconfig.groff...   
    Executing /sbin/conf.d/SuSEconfig.java...    
    Executing /sbin/conf.d/SuSEconfig.kdm...   
    Executing /sbin/conf.d/SuSEconfig.pcmcia...
    Executing /sbin/conf.d/SuSEconfig.perl...
    Executing /sbin/conf.d/SuSEconfig.postfix...
    Executing /sbin/conf.d/SuSEconfig.sendmail...
    Executing /sbin/conf.d/SuSEconfig.susehilf...
    Executing /sbin/conf.d/SuSEconfig.susehilf.add...
    Executing /sbin/conf.d/SuSEconfig.susewm...
    Executing /sbin/conf.d/SuSEconfig.tetex...
    Executing /sbin/conf.d/SuSEconfig.ypclient...
    Processing index files of all manpages...
    Finished.
    

• You also need some scripts to automate startup and shutdown of the Oracle8i listener. The listener is a name server that allows your Oracle programs to talk to local and remote databases using a standard naming convention. It is required for Intermedia Text and full site search.

  Download these three scripts into /var/tmp

  • startlsnr.txt

  • stoplsnr.txt

  • listener8i.txt

  Now issue the following commands (still as root).

  [root ~]# su - oracle
  [oracle ~]$ cp /var/tmp/startlsnr.txt /ora8/m01/app/oracle/product/8.1.7/bin/startlsnr
  [oracle ~]$ cp /var/tmp/stoplsnr.txt /ora8/m01/app/oracle/product/8.1.7/bin/stoplsnr    
  [oracle ~]$ chmod 755 /ora8/m01/app/oracle/product/8.1.7/bin/startlsnr
  [oracle ~]$ chmod 755 /ora8/m01/app/oracle/product/8.1.7/bin/stoplsnr
  [oracle ~]$ exit
  [root ~]# cp /var/tmp/listener8i.txt /etc/rc.d/init.d/listener8i
  [root ~]# cd /etc/rc.d/init.d
  root:/etc/rc.d/init.d# chmod 755 listener8i
  

  Test the listener automation by running the following commands and checking the output.

  root:/etc/rc.d/init.d# ./listener8i stop
  Oracle 8i listener start/stop
  Shutting down Listener for 8i: 
  LSNRCTL for Linux: Version 8.1.7.0.0 - Production on 06-MAR-2002 18:28:49
  
  (c) Copyright 1998, Oracle Corporation.  All rights reserved.
  
  Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost.localdomain)(PORT=1521)))
  The command completed successfully
  
      
  root:/etc/rc.d/init.d# ./listener8i start
  Oracle 8i listener start/stop
  Starting the Listener for 8i: 
  LSNRCTL for Linux: Version 8.1.7.0.0 - Production on 06-MAR-2002 18:28:52
  
  (c) Copyright 1998, Oracle Corporation.  All rights reserved.
  
  Starting /ora8/m01/app/oracle/product/8.1.7/bin/tnslsnr: please wait...
  
  TNSLSNR for Linux: Version 8.1.7.0.0 - Production
  System parameter file is /ora8/m01/app/oracle/product/8.1.7/network/admin/listener.ora
  Log messages written to /ora8/m01/app/oracle/product/8.1.7/network/log/listener.log
  Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=localhost.localdomain)(PORT=1521)))
  Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC)))
  
  Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost.localdomain)(PORT=1521)))
  STATUS of the LISTENER
  ------------------------
  Alias                     LISTENER
  Version                   TNSLSNR for Linux: Version 8.1.7.0.0 - Production
  Start Date                06-MAR-2002 18:28:53
  Uptime                    0 days 0 hr. 0 min. 0 sec
  Trace Level               off
  Security                  OFF
  SNMP                      OFF
  Listener Parameter File   /ora8/m01/app/oracle/product/8.1.7/network/admin/listener.ora
  Listener Log File         /ora8/m01/app/oracle/product/8.1.7/network/log/listener.log
  Services Summary...
    PLSExtProc        has 1 service handler(s)
    ora8      has 1 service handler(s)
  The command completed successfully
  

  This test will verify that the listener is operating normally. Login into the database using the listener naming convention.

  sqlplus username/password/@SID

  [root ~]# su - oracle
  [oracle ~]$ sqlplus system/alexisahunk@ora8
  
  SQL> select sysdate from dual;
  
  SYSDATE
  ----------
  2002-02-22
  
  SQL> exit
  [oracle ~]$ exit
  [root ~]#
  

  • RedHat users:

    Now run chkconfig on the listener8i script.

    [root ~]# cd /etc/rc.d/init.d/
    root:/etc/rc.d/init.d# chkconfig --add listener8i
    root:/etc/rc.d/init.d# chkconfig --list listener8i
    listener8i      0:off   1:off   2:off   3:on    4:on    5:on    6:off
    

  • Debian users:

    Now run update-rc.d on the listener8i script.

    [root ~]# update-rc.d listener8i defaults 21 19
     Adding system startup for /etc/init.d/listener8i ...
       /etc/rc0.d/K19listener8i -> ../init.d/listener8i
       /etc/rc1.d/K19listener8i -> ../init.d/listener8i
       /etc/rc6.d/K19listener8i -> ../init.d/listener8i
       /etc/rc2.d/S21listener8i -> ../init.d/listener8i
       /etc/rc3.d/S21listener8i -> ../init.d/listener8i
       /etc/rc4.d/S21listener8i -> ../init.d/listener8i
       /etc/rc5.d/S21listener8i -> ../init.d/listener8i
    

• Test the automation

  As a final test, reboot your computer and make sure Oracle comes up. You can do this by typing

  [root ~]# /sbin/shutdown -r -t 0 now
  

  Log back in and ensure that Oracle started automatically.

  [joeuser ~]$ su - oracle
  [oracle ~]$ sqlplus system/alexisahunk@ora8
  
  SQL> exit
  

Congratulations, your installation of Oracle 8.1.7 is complete.

ref: https://openacs.org/doc/current/oracle.html

Interfacing with MS Teams and related services (Microsoft Graph)

Created by Gustaf Neumann, last modified by Gustaf Neumann 25 Mar 2022, at 02:14 PM

The xooauth package support a basic interface for the Microsoft Graph API, which can be used e.g. with Microsoft Teams.

===========================================================================

ms::app pp ?-list? ?-prefix /value/? /dict/

ms::app application get /application_id/ ?-select /value/?
ms::app application list ?-count /value/? ?-expand /value/? ?-filter /value/? ?-orderby /value/? ?-search /value/? ?-select /value/? ?-top /integer/?

ms::app chat get /chat_id/
ms::app chat messages /chat_id/ ?-top /integer/?

ms::app group deleted ?-count /value/? ?-expand /value/? ?-filter /value/? ?-orderby /value/? ?-search /value/? ?-select /value/? ?-top /integer/?
ms::app group get /group_id/ ?-select /value/?
ms::app group list ?-count /value/? ?-expand /value/? ?-filter /value/? ?-orderby /value/? ?-search /value/? ?-select /value/? ?-max_entries /value/? ?-top /integer/?
ms::app group member add /group_id/ /principals/
ms::app group member list /group_id/ ?-count /value/? ?-filter /value/? ?-search /value/? ?-max_entries /value/? ?-top /integer/?
ms::app group member remove /group_id/ /principal/
ms::app group memberof /group_id/ ?-count /value/? ?-filter /value/? ?-orderby /value/? ?-search /value/?
ms::app group owner add /group_id/ /principal/
ms::app group owner list /group_id/
ms::app group owner remove /group_id/ /user_id/

ms::app team archive /team_id/ ?-shouldSetSpoSiteReadOnlyForMembers /value/? ?-donecallback /value/? ?-wait?
ms::app team channel list /team_id/ ?-filter /value/? ?-select /value/? ?-expand /value/?
ms::app team clone /team_id/ ?-classification /value/? ?-description /value/? -displayName /value/ ?-mailNickname /value/? -partsToClone /value/ ?-visibility /value/? ?-donecallback /value/? ?-wait?
ms::app team create ?-description /value/? -displayName /value/ ?-visibility /value/? -owner /value/ ?-donecallback /value/? ?-wait?
ms::app team delete /team_id/
ms::app team get /team_id/ ?-expand /value/? ?-select /value/?
ms::app team member add /team_id/ /principal/ ?-roles /value/?
ms::app team member list /team_id/ ?-filter /value/? ?-select /value/?
ms::app team member remove /team_id/ /principal/
ms::app team unarchive /team_id/ ?-donecallback /value/? ?-wait?

ms::app user get /principal/ ?-select /value/?
ms::app user list ?-select /value/? ?-filter /value/? ?-max_entries /value/? ?-top /value/?
ms::app user me ?-select /value/? ?-token /value/?
ms::app user memberof /principal/ ?-count /value/? ?-filter /value/? ?-orderby /value/? ?-search /value/?

ms::app run_donecallback /location/ /callback/
ms::app schedule_donecallback /secs/ /location/ /callback/
ms::app token ?-grant_type /value/? ?-scope /value/? ?-assertion /value/? ?-requested_token_use /value/?

Example Usage

#
# Create the interface object for a tenant (named here ms::app).
# For interacting with multiple tenant, define multiple application
# interface objects.
#
::ms::Graph create ms::app \
    -tenant ... \
    -client_id ... \
    -client_secret ... \
    -version v1.0

#
# get the Teams UID for a user (here via email, actual userPrincipalName)
#
set user_info [ms::app user get gustaf.neumann@wu.ac.at]
set user_id [dict get $user_info id]

:  ba34495a-fd40-4c82-bc7b-1f7c778fec34

#
# Get information about a user. We use for output formatting a
# pretty-printer to provide a more readable format pf the dict
# structures returned by the Microsoft graph API:
#
ms::app pp [ms::app user get gustaf.neumann@wu.ac.at]

: @odata.context: https://graph.microsoft.com/v1.0/$metadata#users/$entity
: businessPhones: {...}
: displayName: Neumann, Gustaf
: givenName: Gustaf
: jobTitle: null
: mail: Gustaf.Neumann@wu.ac.at
: mobilePhone: ....
: officeLocation: D2.2.034
: preferredLanguage: null
: surname: Neumann
: userPrincipalName: gustaf.neumann@wu.ac.at
: id: ba34495a-fd40-4c82-bc7b-1f7c778fec34

#
# One can get more information by specifying additional "select"
# attributes, such as e.g. "department" and others (for details, see
# https://docs.microsoft.com/en-us/graph/api/resources/user?view=graph-rest-1.0#properties)

ms::app pp [ms::app user get gustaf.neumann@wu.ac.at -select id,department,companyName,identities,mySite,streetAddress]

: @odata.context: https://graph.microsoft.com/v1.0/$metadata#users(id,department,companyName,identities,mySite,streetAddress)/$entity
: id: ba34495a-fd40-4c82-bc7b-1f7c778fec34
: department: Wirtschaftsinformatik und Neue Medien
: companyName: WU-WIEN
: streetAddress: Welthandelsplatz 1
: mySite: https://wu-my.sharepoint.com/personal/gustaf_neumann_wu_ac_at/
: identities: {signInType userPrincipalName issuer wu.onmicrosoft.com issuerAssignedId gustaf.neumann@wu.ac.at}

#
# Get a list of certain users. We use for output formatting a
# pretty-printer to provide a more readable format of the dict
# structures returned by the Microsoft graph API:
#
ms::app pp [ms::app user list -select id,displayName,userPrincipalName -filter "startsWith(displayName,'Neumann')"]

: @odata.context: https://graph.microsoft.com/v1.0/$metadata#users(id,displayName,userPrincipalName)
: value:
:    id: ba34495a-fd40-4c82-bc7b-1f7c778fec34
:    displayName: Neumann, Gustaf
:    userPrincipalName: gustaf.neumann@wu.ac.at
:
:    id: 4e2b2b37-6c50-4367-9209-bd7392f2e115
:    displayName: Neumann, Lore
:    userPrincipalName: lore.neumann@wu.ac.at

#
# Return the first 10 users. Per default, Microsoft Graph returns the
# first 100 entries. By specifying -max_entries, the interface issues
# potentially several requests for returning the desired amount.
#
ms::app pp [ms::app user list -select id,displayName,userPrincipalName -max_entries 10]

#
# List the first 10 teams/groups
#
ms::app pp [ms::app group list -select id,displayName -max_entries 10]

#
# Get some attributes about a set of teams
#
ms::app pp [ms::app group list -select id,displayName -filter "startsWith(displayName,'TLF')"]

:
: @odata.context: https://graph.microsoft.com/beta/$metadata#groups(id,displayName)
: value:
:    id: b78e7642-...
:    displayName: TLF-TEAM

#
# Get detail info from a team
#
set team_id b78e7642-...
ms::app pp [ms::app team get $team_id]

#
# Get members of a team
#
ms::app pp [ms::app group member list $team_id]

#
# Get owners of a team
#
ms::app pp [ms::app group owner list $team_id]

#
# Add member to a team
#
ms::app group member add $team_id gustaf.neumann@wu.ac.at

#
# Get channels of a team
#
ms::app pp [ms::app team channel list $team_id]

#
# Delete a team
#
ms::app team delete 85f3d2d2-c2d3-...

#
# List deleted groups/teams
#
ms::app pp [ms::app group deleted -filter "startsWith(displayName,'Testing community')"]

: @odata.context: https://graph.microsoft.com/v1.0/$metadata#groups(id,displayName,deletedDateTime)
: value:
:     id: c0030714-656d-4bbe-8d4e-507e73d6f643
:     displayName: Testing community 3
:     deletedDateTime: 2021-10-22T09:14:08Z

New Interface for Calling Database functions

Created by Gustaf Neumann, last modified by Gustaf Neumann 10 Mar 2022, at 09:34 AM

Scheduled for OpenACS 5.10.1, there is a new interface available that provides several improvements over the current solutions:

The new interface ...

• is significantly faster than the existing official OpenACS interface (package_exec_plsql) and easier to use by supporting a standard calling interface (not the special "var_list" which has to be assembled for package_exec_plsql)
• is about the same performance as the "::xo::db::sql::*" interface without sharing its disadvantages (being a separate pattern, just one backend, ...)
• is more secure (thorough argument checking on the Tcl and bind-vars level), and
• more flexible (works with multiple databases and multiple driver types), and
• much more feature-complete (e.g. support for functions returning tables, etc.)

For more details, check below. The plan is to replace the "::xo::db::sql::*" interface in a first step and replace calls to package_exec_plsql in the supported packages, and to mark the obsolete functions as deprecated. In the near future, more features of the xo::db interface will be added. The plain ugly old interface of OpenACS of calling db-functions where the highest priority.

Implemented features:

• SQL function can be called for multiple database connection types (driver types nsdb and nsdbi, backend types PostgreSQL and Oracle) with less overhead (in essence, a value added, better performing replacement of "package_exec_plsql"). The  interface provided via xo::db supported just a single combination of the above.

• Support for PostgreSQL and Oracle

• Ability to talk to multiple databases from the same OpenACS instance. These databases can be

  • multiple databases of the same driver and backend  (e.g. multiple PostgreSQL databases)
  • multiple databases using different drivers (e.g. nsdb and nsdbi)
  • multiple databases with different backends vendors (e.g. PostgreSQL and Oracle).

  The selection of the backed happens of via the standard "-dbn" parameter. For the nsdb driver, one can use  e.g., "-dbn legacy", an example for nsdbi is "-dbn dbi1",  where the value provided via "-dbn" is passed for uniformity to the "-db" option of the nsdbi API.

• Support for functions returning composite SQL types (SQL type "record" in PostgreSQL or "table" in Oracle). Results are returned as lists of lists.  This feature is implemented and tested for PostgreSQL connected via nsdb and nsdbi drivers and Oracle via nsdb.

• SQL functions returning "void" can be called as well. In the Oracle cases, these are "procedures" which have to be called differently.

• Additional (Tcl-level) argument checking is provided before calling into SQL for improved security and documentation. The API handles among other types the SQL types "integer" and "bigint".

• Arguments of SQL functions are passed to the database via bind variables (implemented for PostgreSQL with nsdb and nsdbi). This is a security improvement over the previous  implementation in xotcl-core.

• Standard default handling (argument default values like for all Tcl procs, although the way how defaults are handled is different in PostgreSQL and Oracle).

• Optional arguments passed in as empty strings are treated as NULL values (standard behavior in OpenACS)

• Dropped shortcomings of xo::db interface (naming and calling conflicts).

• Independent of xotcl-core

 Possible further extensions:

• Argument types could/should be displayed in the API browser (general feature request, not specific to the new DB interface)

Usage:

At startup, a single database interface is creates, which is named "::acs::dc", which takes the parameters from the default  setup of the OpenACS configuration.

 The following command creates a second database interface based on the "nsdbi" driver and define the interface stubs for the nsdbi driver.

::acs::db::require_dc -driver nsdbi -name ::acs::dc2
::acs::dc2 create_db_function_interface

For specifying a different backend, one could use e.g.:

::acs::db::require_dc -backend oracle -name ::acs::dc3

SQL query to a second DB (db pool called "legacy") via nsdb interface:

::acs::dc list_of_lists -dbn legacy get_list {select 1 from dual}

 Call database function from a second DB (db pool called "legacy") via nsdb driver:

::acs::dc call content_item get_latest_revision -dbn legacy -item_id 33357

 Call database function from a second DB via nsdbi driver named "dbi1"

::acs::dc2 call content_item get_latest_revision -dbn dbi1 -item_id 33357

 Return multiple records via nsdb

::acs::dc call content_item get_children -item_id 169303

 Return multiple records via nsdbi

::acs::dc2 call content_item get_children -item_id 169303

 Some calls with performance data (for ds/shell):

lappend _ [package_exec_plsql -var_list {{item_id 33357}} content_item get_latest_revision]
lappend _ [::xo::db::sql::content_item get_latest_revision -item_id 33357]
lappend _ [::acs::dc call content_item get_latest_revision -item_id 33357]
lappend _ [::acs::dc2 call content_item get_latest_revision -item_id 33357]

lappend _ [time {package_exec_plsql -var_list {{item_id 33357}} content_item get_latest_revision} 1000]
lappend _ [time {::xo::db::sql::content_item get_latest_revision -item_id 33357} 1000]
lappend _ [time {::acs::dc call content_item get_latest_revision -item_id 33357} 1000]
lappend _ [time {::acs::dc2 call content_item get_latest_revision -item_id 33357} 1000]
join $_ \n

The results as displayed by ds/shell

33358
33358
33358
33358
402.096544 microseconds per iteration
190.805835 microseconds per iteration
187.885927 microseconds per iteration
171.99336300000002 microseconds per iteration

Server-sent events

Created by Gustaf Neumann, last modified by Gustaf Neumann 14 Jan 2022, at 09:29 PM

Server-Sent Events (SSE) is a server push technology enabling a client to receive automatic updates from a server via an HTTP connection, and describes how servers can initiate data transmission towards clients once an initial client connection has been established.

A simple approach for implementing SSE on OpenACS with discussed in a forum thread.

The example below shows, how to use SSE to associate a background job with the client. This can be used e.g. when executing longer a running job in the background and to keep the client up incrementally to date what is currently happening. This can be seen as an alternative to streaming HTML.

Page with associated background activity

ad_page_contract {

    Sample page for emulating streaming HTML via SSE (server side
    events) for a background job. In this example, the same page is
    used as the event sink (where the events are displayed) and as an
    event source (when called with a session_id).

    @author Gustaf Neumann
} {
    {session_id:integer ""}
}

set title "Sample HTML streaming job page (SSE)"
set context $title

if {$session_id ne ""} {
    #
    # We are called by the event handler from JavaScript. This block
    # could be a different, generic script, but we keep it here
    # together for reducing number of files.
    #

    #
    # Set up SSE: return headers and register reporting channel for
    # the session_id.
    #
    set channel [ns_connchan detach]
    ns_connchan write $channel [append _ \
                                    "HTTP/1.1 200 OK\r\n" \
                                    "Cache-Control: no-cache\r\n" \
                                    "X-Accel-Buffering': no\r\n" \
                                    "Content-type: text/event-stream\r\n" \
                                    "\r\n"]
    sse::channel $session_id $channel
    ad_script_abort
} else {
    #
    # We are called as an ADP page.
    #
    set session_id [ns_conn id]

    #
    # Register the SSE event handler in JavaScript.
    #
    template::head::add_script -script [subst {
        if(typeof(EventSource) !== "undefined") {
            var sse = new EventSource("[ns_conn url]?session_id=$session_id");
            sse.onmessage = function(event) {
                if ('__CLOSE_SSE__' == event.data) {
                    sse.close(); // stop retry
                } else {
                    document.getElementById("result").innerHTML += event.data;
                }
            };
        } else {
            document.getElementById("result").innerHTML = "Sorry, your browser does not support server-sent events...";
        }
    }]

    #
    # Run some job in the background and report updates via SSE to the
    # current page. The session_id is used to associate the background
    # job with the client.
    #
    sse::job $session_id {
        foreach i {1 2 3 4} {
            set HTML "<li>finish $i: ...([person::name -person_id [ad_conn user_id]])</li>"
            sse::message -localize $session_id $HTML
            ns_sleep 1s
        }
        sse::close $session_id
    }

    ad_return_template
}

Corresponding ADP page:

@title;literal@
@context;literal@
<h1>Getting updates from job associated with session_id @session_id@</h1> 
<ul> </ul>

Library support

Save this under e.g. acs-tcl/tcl/sse-procs.tcl

#############################################################################
# Simple API for SSE messages via a SESSION_ID, e.g. via a job running
# in the background.
#
# Gustaf Neumann fecit
#############################################################################
namespace eval sse {

  #
  # sse::job
  #
  ad_proc ::sse::job { {-lang} session_id script} {
    
    Execute some (long running) script in the background.  The
    background script is associated via the session_id with the client
    and can send messages to it via sse::message. When done, the job
    should issue sse::close.
    
    @param lang language used for internationalization. If not set,
           taken from the calling environment
    @param session_id associated session_id, used for reporting back
    @param script script to be executed in the background
    
  } {
    if {![info exists lang]} {
      set lang [uplevel {ad_conn locale}]
    }
    nsv_set sse_dict $session_id \
        [list \
             lang $lang \
             ad_conn [list array set ::ad_conn [array get ::ad_conn]] \
             xo::cc [expr {[info commands ::xo::cc] ne "" ? [::xo::cc serialize] : ""} ] \
            ]
    ns_job queue -detached sse-jobs \
        [list ::apply [list session_id "sse::init_job $session_id\n$script"] $session_id]
  }

  #
  # sse::message
  #
  ad_proc ::sse::message { {-localize:boolean} session_id msg} {

    Send a message to the client. In case the channel
    was not jet registered, buffer the message in an nsv dict.
    
    @param localize Perform localization of the provided text
    @param session_id associated session_id, used for associating output session
    @param msg The message to be sent back
  } {
    if {$localize_p} {
      set msg [lang::util::localize $msg [nsv_dict get sse_dict $session_id lang]]
    }
    
    if {[nsv_dict get -varname channel sse_dict $session_id channel]} {
      #
      # The channel is already set up.
      #
      if {[nsv_dict get -varname messages sse_dict $session_id messages]} {
        #
        # The "nsv_dict unset" poses a potential race condition,
        # ... but not for the job application scenario.
        #
        nsv_dict unset sse_dict $session_id messages
        #ns_log notice "--- sse $session_id get buffered messages '$messages'"
      } else {
        set messages ""
      }
      
      lappend messages $msg
      foreach message $messages {
        ns_log notice "--- sse $session_id message '$message'"
        #
        # SSE needs handling of new-lines in the data field; here we
        # send two messages in this case. In other cases, maybe coding
        # newline as literally \n and decoding it on the client might
        # be appropriate.
        #
        ns_connchan write $channel \
            [string cat "data:" [join [split $message \n] "\ndata:"] "\n\n"]
        
        if {$message eq "__CLOSE_SSE__"} {
          ns_connchan close $channel
          nsv_unset sse_dict $session_id 
        }
      }
    } else {
      #
      # Buffer the message.
      #
      #ns_log notice "--- sse $session_id must buffer message '$msg'"
      nsv_dict lappend sse_dict $session_id messages $msg
    }
  }
  
  #
  # sse::channel: register a channel for the session id. This is
  # typically called by the event handler, which is called from
  # JavaScript.
  #
  proc ::sse::channel {session_id channel} {
    nsv_dict set sse_dict $session_id channel $channel
  }

  #
  # sse::init_job: setup a context quite similar to a connection thread
  #  
  proc ::sse::init_job {session_id} {
    set sse_dict [nsv_get sse_dict $session_id]
    eval [dict get $sse_dict ad_conn]
    eval [dict get $sse_dict xo::cc]    
  }
  
  #
  # sse::close: provide an API for closing the session
  #
  proc ::sse::close {session_id} {
    ::sse::message $session_id __CLOSE_SSE__
  }
  
  if {"sse-jobs" ni [ns_job queues]} {
    ns_job create sse-jobs 4
    nsv_set sse_dict "" ""    
    #ns_job configure -jobsperthread 10000
  }
}
#
# Local variables:
#    mode: tcl
#    tcl-indent-level: 2
#    indent-tabs-mode: nil
# End:

Im case you are running NGINX and you are experiencing problems with SSE, check out this entry on stackoverflow.

Theming in OpenACS

Created by Gustaf Neumann, last modified by Paul Babin 14 Oct 2021, at 04:01 PM

OpenACS provides theming functionality, i.e. the customization of the site layout through so-called theme packages with customized CSS and HTML layout.

Each subsite in OpenACS can have its own theme.

By default, the "Openacs-Default-Theme" is activated when installing OpenACS. The "Openacs-Default-Theme" package contains a "tabbed" navigation theme and a "plain" navigation theme (without tabs in main navigation).

OpenACS comes now with a new theme package, the "Openacs-Bootstrap3-Theme" Package, which is based on the Twitter Bootstrap Library. This new package is used on the openacs.org site and available form the openacs repository. The package (and the description below) requires at least OpenACS 5.9.0.  In the following section you find some hints on how to use and customize this theme and how the theming can be customized in newer versions of OpenACS.

Step-by-Step Installation of a Theme Package

Installing theme packages and switching between themes can be easily done in a few steps:

1. Obtain the theme package (e.g. from CVS or github) and install it under the package directory. You can as well "install from the respository", which makes step 2 below unnecessary.
2. Go to /acs-admin/install and install the package
3. Navigate to the subsite whose theme you would like to change and go to the "Configure"-Page. If you only have the default subsite set up, this is located at /admin/configure. Choose the theme you want. Click Ok. You should see the new theme now.
   Note that a theme package can contain a number of different themes. For instance, the "Openacs-Default-Theme" Package contains a "tabbed" and a "plain" theme; the "Openacs-Bootstrap3-Theme" Package contains a "tabbed" theme in OpenACS style and a "tabbed" theme in turquoise color.
4. DONE!

Understanding the Basic Settings of Theme Packages

Where are the themes of a theme package defined?

Open /packages/openacs-bootstrap3-theme/tcl/apm-callback-procs.tcl in your editor. You see that 2 themes are defined here (subsite::new_subsite_theme), with all relevant parameters including css and js files and definition of template files. 

You could add another subsite theme here. In order to get it listed on the "Configure"-Page of your subsite, you need to de-install and re-install the Theme Package.

The Theming Parameters allow you to specify, how elements of your page such as forms or lists should be rendered. If the resource which you have defned in your theme is not found, the default is taken from /packages/acs-templating/resources/ as a fallback.

The Theming Parameters

Have a look at the parameter page of your subsite. If you only have the default subsite set up, this is located at /shared/parameters.

In the "Theming"-Section of the subsite parameter page, you find the parameters which you have defined in the apm-callback-procs.tcl file of your theme package. You can modify the parameters here, however it is recommended that you define them in the apm-callback-procs. Why? If you switch the theme or re-install it, the settings of the parameter page are lost!

DefaultDimensionalStyle

• Defines layout of the filter options in a dimensional list. You can find an example at the Download Page of the OpenACS Website.

DefaultFormStyle

• Defines the layout of form templates

DefaultListStyle

• Defines the layout of list templates

DefaultMaster

• Defines the general page layout. The master usually sets the framing for the page layout and places elements such as header, navigation, content area, footer.

ResourceDir

• The directory, from which the resources of acs-templating are loaded. The resources include the template files for dimensionals, forms, lists, masters, templates, widgets. If not set, or the required resource is not found, the default is taken from /packages/acs-templating/resources/.
  If for instance, the ResourceDir is set to /packages/openacs-bootstrap3-theme/resources, and you have a dimensional template defined in "Openacs-Bootstrap3-Theme" Package, you only need to specify the name of the -dimensional_template without giving the entire path.

StreamingHead

• Master used for streaming content

ThemeCSS

• List of CSS files used in the theme

ThemeJS

• List of Javascript files used in the theme

ThemeKey

• Key of theme.

Additional Tools to Layout in a Theme Package

Widgets

Starting with OpenACS 5.9, so-called widgets can be defined. <widget> is very similar to <include>, but uses widget specific name resolution based on themes. If the theme package contains resources/widgets/$specifiedName it is used from there. Otherwise it behaves exactly like <include> (without the resources/templates/theming).

In the "Openacs-Bootstrap3-Theme" Package, widgets are defined e.g. for the main navigation bar and the login area.

Templates

Sometimes, you might want a specific page or include to have a special design or layout which differs from the standard layout defined in that package. For instance, you might want to give the notification widget a new look&feel. You could go to the notification package and edit the correspondend tcl/adp files there. However, when upgrading the package, you always would have to merge your local changes with the upgraded files.

Starting with OpenACS 5.9, you can customize your local templates in your Theme Package. These files "overrule" the tcl/adp files in the corresponding packages. This makes it easier for you to keep the overview of all your layout changes, and also to upgrade your packages with less effort.

How does it work?

Let's take the example from above: you want to change the lokk&feel of the notification widget. The tcl/adp files are situated in /packages/notifications/lib/notification-widget.tcl and /packages/notifications/lib/notification-widget.adp.

Go to the "Openacs-Bootstrap3-Theme" Package. You see a folder resources/templates. Now construct a folder structure in the templates folder, just like this: resources/templates/notifications/lib and create the 2 files notification-widget.tcl/adp. Change the layout as desired.

DONE!

The "Openacs-Bootstrap3-Theme" Package contains a few such template files by default; feel free to change or complement them.

Understanding the Openacs-Bootstrap3-Theme Package

Different master files

The "Openacs-Bootstrap3-Theme" Package comes with a set of master files, which are shortly explained here. 

• The "base master" is the plain-master.tcl/adp. It defines the general page layout.
• The tabbed-master.tcl/adp is the addition for a tabbed navigation, i.e. navigation tabs in the header bar which are defined in the parameter page. Tabs can be defined for admins (AdminNavbarTabsList), for members (MembersViewNavbarTabsList), for users (UserNavbarTabsList). You can check if you want the applications to be shown as tabs in the navigation bar.
• The tabbed-master-turquois.tcl/adp is similar tp the tabbed-master.tcl/adp. The difference is in the logo image defined in the .tcl file. In the OpenACS website, it is currently applied in the subsite for the .LRN project.
• The plain-streaming-head.tcl/adp, tabbed-streaming-head.tcl/adp, and tabbed-streaming-head-turquois.tcl/adp files are the corresponding master files for streaming content.

Color css

"Openacs-Bootstrap3-Theme" Package contains a "tabbed" theme in OpenACS style and a "tabbed" theme in turquoise color. The color stylesheets are defined in the apm-callback-procs.tcl file and located at /packages/openacs-bootstrap3-theme/www/resources/css/color/. This makes it very easy to change the color design of the theme. You could either modify one of the color stylesheets "blue.css" or "turquois.css"; or you could add your own color stylesheet and link it in the apm-callback-procs.tcl-file. In the latter case, you should de-activate and re-activate the theme in the "Configure"-Page (see above) to activate the setting.

Twitter Bootstrap

As the name of the package suggests, the "Openacs-Bootstrap3-Theme" Package makes use of the Twitter Bootstrap Library in version 3. The Twitter Bootstrap Library is a HTML, CSS and JAVASCRIPT library for frontend development which is based on a mobile first, responsive approach to web design.

Much has been written about this library. However, here are a few links which should help you to make yourself familiar with the Library in case you are new to Bootstrap:

• http://getbootstrap.com/getting-started/: Really helpful to get familiar with the notation specialties.
• https://en.wikipedia.org/wiki/Bootstrap_(front-end_framework): Some background information
• https://github.com/twbs/bootstrap: Bootstrap on GitHub
• https://jelvix.com/blog/bootstrap-vs-material: Compare Bootstrap with Google's Material Design

Designing your own Theme

 The following guideline helps you to create your own theme. The easiest approach to develop your own theme is to build up on an existing one. For instance, you could modify the "Openacs-Bootstrap3-Theme" Package as a starting point.

1. Make a local copy of the "Openacs-Bootstrap3-Theme" Package
2. Rename the package.
3. Take particular care of the .info file. Rename all the places where you find "openacs-bootstrap3-theme", and give a unique package key.
4. Take particular care of the tcl/apm-callback-procs.tcl file. The subsites defined here through subsite::new_subsite_theme need to have a unique key, so give all the subsites you create here a unique key. Also, take care of the namespace.
5. Before starting to modify the package, it is best to try to install and de-install the package on a test instance to make sure that you have all the naming right.
6. Now feel free to modify the design as you like.

Themes in .LRN

When installing .LRN, the theme-zen is installed and activated by default.

.LRN comes now with a new theme package, the "Dotlrn-Bootstrap3-Theme" Package (as of July 2016, it is in proof-of-concept state!).

The "Dotlrn-Bootstrap3-Theme" Package builds on the "Openacs-Bootstrap3-Theme" Package and depends on both the "Openacs-Bootstrap3-Theme" Package and the "Theme-Zen" Package.

The Theming specifics described above also apply to .LRN theming, but additionally you should consider the following points:

Portal Pages Design

The Design of a Portal Page is defined on 3 levels:

• The subsite theme (which can be chosen on the "Configure"- Page, as described above)
• The page layout: It defines whether the page has, for instance, a 1-column, 2-column or 3-column layout. In the "Dotlrn-Bootstrap3-Theme" Package, the layouts are defined at lib/layouts.
• The portlet theme: It defines the look&feel of the single portlet. In the "Dotlrn-Bootstrap3-Theme" Package, the themes are defined at lib/themes.

Since every portal which is created, stores the portlet theme ID in the database, and since every portal page that is created, stores the page layout ID in the database, changing a theme in .LRN requires an upgrade of the affected database tables. This is done by the tcl/theme-procs.tcl/xql in the "Dotlrn-Bootstrap3-Theme" Package.

!!!!!! Attention: If you switch to the .LRN-bootstrap3-theme, all the portal pages are updated to the bootstrap3 layout !!!!!!

Installing and De-Installing the "Dotlrn-Bootstrap3-Theme" Package

If the "Dotlrn-Bootstrap3-Theme" Package is de-installed, the above described portal pages design falls back to the theme-zen design. This should make sure that the portal pages are always provided with a working layout.

Upgrade from PostgreSQL 9.6 to 13.1

Created by Gustaf Neumann, last modified by Gustaf Neumann 08 Oct 2021, at 09:57 PM

Here is a summary of steps to upgrade a legacy OpenACS installation (recent OpenACS, but old PostgreSQL version). In general it is possible to restore a "pg_dump" of some older version in newer versions of PostgreSQL. It is not necessary, to upgrade in small steps, version by version, but one can also skip major versions, when keeping certain things in mind. The upgrade on openacs.org was from PostgreSQL 9.6.4 to 13.1.

OIDs

Newer versions of PostgreSQL do not support OIDs anymore. OpenACS does not need OIDs in its queries, but one has to create a pg_dump without OIDs first. This can be achieved by the following commands.

 #
 # Settings of old PostgreSQL installation
 #
 PGBIN=/usr/local/pg964/bin
 PGPORT=5496

 #
 # Database from where OIDs should be removed
 #
 DB=openacs.org

 for tbl in `$PGBIN/psql -p $PGPORT -U postgres -qAt -c "select schemaname || '.' || tablename from pg_tables WHERE schemaname <> 'pg_catalog' AND schemaname <> 'information_schema';" $DB`
 do
    $PGBIN/psql -U postgres -c "alter table $tbl SET WITHOUT OIDS" $DB
 done

tsearch2

tsearch2 was an extension in PostgreSQL versions before version 10, but was dropped as an extension at this time, being now integral part of newer PostgreSQL versions. Users who have been continuously upgrading since a long time (from before 8.3) have to manually modify their databases by stripping leftovers from the extension. Otherwise, loading the dump will raise errors doe to missing tsearch2 functions in new versions (e.g. "ERROR: function "dex_init(internal)" does not exist").

By excluding the "pg_ts_*" table, the dump can be imported (although it will complain about the missing tsearch2 module). Nevertheless, the imported dump will be fully functional. 

Dump from PostgreSQL 9.6

Here is, what i've used on openacs.org for dumping. For your local installation, you have to replace the database names in the for loop by the names that you use.

  # 
  # Use variable DATE to disambiguate multiple dumps in the same directory 
  #
  DATE=`date +"%Y-%m-%d"`

  for DB in 5-9-1 dotlrn-test fisheye oacs-5-8 openacs.org openacs.org-test translate wi1.org wi1.org-test
  do
    echo "dumping to ~/$DB.dump.$DATE.gz"
    $PGBIN/pg_dump -p $PGPORT -U postgres --exclude-table=public.pg_ts_\* $DB | gzip > ~/$DB.dump.$DATE.gz
  done

Restore in PostgreSQL 13

After this, restoring went smooth:

  #
  # Adjust settings to your pg13 installation
  #
  export PG13=pg13
  export PGBIN13=/usr/local/${PG13}/bin
  export PGPORT13=5413

  for DB in 5-9-1 dotlrn-test oacs-5-8 openacs.org openacs.org-test translate wi1.org  wi1.org-test
  do
    echo "reloading from ~/$DB.dump.$DATE.gz"
    $PGBIN13/dropdb -p $PGPORT13 -U nsadmin $DB
    $PGBIN13/createdb -p $PGPORT13 -U openacs -T template0 $DB
    $PGBIN13/psql -p $PGPORT13 -U openacs $DB -tAc "create extension ltree"
    gunzip < ~/$DB.dump.$DATE.gz | $PGBIN13/psql -p $PGPORT13 -U nsadmin $DB > ~/$DB.log
  done

You might not need the line with the ltree extension in your installation.

Upgrade from PostgreSQL 13 to PostgreSQL 14

Just dump and restore.

Upgrade to OpenACS 5.10

Created by Gustaf Neumann, last modified by Gustaf Neumann 07 Oct 2021, at 01:57 PM

Upgrades from earlier version than OpenACS 5.8 should read upgrade-oacs-5-8 and upgrade-oacs-5-9.

OpenACS 5.10 requires PostgreSQL 9.6 or newer, Tcl 8.6.2 or newer and XOTcl 2.1 or newer (part of the nsf package). XOTcl 2.0 can be installed e.g. via install-ns, or from Debian sid https://packages.debian.org/sid/, or from sources https://next-scripting.org/. In order to check the versions of these packages already installed in your OpenACS installation, check the output of /xotcl/version-numbers in your installation.

Before upgrading to OpenACS 5.10, upgrade all packages in your current installation to recent versions in your OpenACS 5.9 installation, which OpenACA 5.9.1(i.e. in the oacs-5-9 channel).

In general it is always recommended to backup your current installation before overwriting it. Make e.g. a tar archive of all files of the OpenACS tree and a database dump.

After upgrading the packages in the oacs-5-9 channel,  the following steps are recommended:

• Restart the server with the newest OpenACS 5.9
• Install newest version of the source files (e.g. from the OpenACS 5.10 tar distributions, or install/upgrade from the oacs-5-10 branch from cvs),
• Upgrade the acs-core packages via package manager, restart server
• Upgrade/install application packages via package manager as needed (from file-system or from repository)

Note: when one installs now from the openacs-5.9.1 tar ball, there will be a problem with a PostgreSQL 11, since the tar ball is older than this PostgreSQL release. However, the version of OpenACS 5.9.1 in the repositories (CVS or github) can be installed as well with newer PostgreSQL versions.

Step by step upgrade to OpenACS 5.10.0

The following steps assume, you have a plain, not modified version of OpenACS running.

1. Upgrade to the latest version of OpenACS 5.9.1, e.g. via
      cvs -r oacs-5-9 openacs-4
   upgrade the packages, restart the server and make sure, you have really upgraded all packages;
   under /acs-admin/apm/ you should see a Kernel like 5.9.1* running.
    
2. Either fet release tar file of openacs-5.10.0, unpack it (e.g. under /usr/local), make sure, your configuration file points to the right source tree, and restart server, or upgrade via cvs
      cvs -r oacs-5-10 -Pd openacs-4
    
3. Upgrade packages:
   Browse to /acs-admin/install click on "upgrade from local", select all packages, restart server - done.

In case, you have a locally modified version of OpenACS, but you have not modified the kernel packages, then make sure to get the tar file of the core packages in step (2), proceed as described, upgrade the acs-core packages first and then the application packages of your installation.

In case you have in your installation as well changes in the acs-core packages (which is not recommended, but it happens), then get the diffs (you local changes) between the acs-core packages of oacs-5-10 and your installation, proceed as with the unmodified acs-core packages (paragraph above) and apply/merge your local changes after upgrading.

These steps are working also with recent versions of PostgreSQL such as 13.3.

xowf (XoWiki Workflow)

Created by Gustaf Neumann, last modified by Gustaf Neumann 02 Oct 2021, at 10:41 AM

XoWiki Content Flow - an XoWiki based workflow system implementing state-based behaviour of wiki pages and forms (short intro).

Package Specification Summary for Package: xowf

Bug Tracker Summary for Package: xowf

There is no package with the name "xowf" known to bug-tracker.

Code Metrics Summary for Package: xowf

xowf depends on xowiki.

Workflow Examples:

• A two-chambers legislative process, loosely based on the Italian one. A law proposal is submitted to the first chamber, then passed to the second one. The text keeps being passed between the two chambers until no amendments (changes) are made. Once this condition is met, the law is approved. Workflow example of a legislative process

Streaming HTML

Created by Gustaf Neumann, last modified by Gustaf Neumann 11 Sep 2021, at 09:58 AM

Streaming HTML can be used to output HTML content to a user incrementally. This is in particular useful for pages for pages with longer response time, to inform during processing about the progress of the tasks.

Newer OpenACS versions come with templates for the standard themes for streaming pages. The template for streaming pages can be retrieved for the current theme via the API call "template::streaming_template". An application developer should structure streaming HTML pages as follows:

Output top of page:

set context ...
set title ...
ad_return_top_of_page [ad_parse_template \
    -params [list context title] \
    [template::streaming_template]]

Output HTML incrementally:

# Assume, we are collecting HTML in the Tcl variable HTML.
# Send this HTML chunk incrementally to the user
ns_write [lang::util::localize $HTML]

Flush Output from body scripts:
(e.g. template::add_body_script, template::add_event_listener, template::add_body_handler, template::add_script)

ns_write [template::collect_body_scripts]

End of Page:

# Optionally
ns_write [lang::util::localize [template::get_footer_html]]
ns_write [template::collect_body_scripts]

Full sample script:
Putting everything together

set title "Sample HTML streaming page"
set context [list $title]
ad_return_top_of_page [ad_parse_template \
                           -params [list context title] \
                           [template::streaming_template]]
ns_write "<ul>\n"
foreach i {1 2 3 4} {
    set HTML "<li>finish $i: ...</li>\n"
    ns_write [lang::util::localize $HTML]
    ns_write [template::collect_body_scripts]
    ns_sleep 1s
}
ns_write "</ul>\n"
ns_write "<p>Done.</p>"

ns_write [lang::util::localize [template::get_footer_html]]
ns_write [template::collect_body_scripts]

See this sample script in action: https://openacs.org/streaming

Caveat: Windows PCs having the current (Sept 2020) version of Bitdefender (antivirus software ) installed with HTTP traffic scanning activated don't show the incremental rendering of the page, but just the full page after it has finished. This might be seen as a problem of Bitdefender and affects streaming HTML from all sites.

Available OpenACS Packages

Created by Gustaf Neumann, last modified by Gustaf Neumann 03 Sep 2021, at 12:00 PM

Wiki pages for the packages available in the OpenACS code repository:

List of the actively maintained packages

In the oacs-5-10 channel, currently 93 packages are actively maintained. Additionally, the packages is use on openacs.org are maintained as well, but not necessarily recommended for new installations. The packages in the main branch are in different states, some of these work perfecty, some of these migh use outdated calls an can be seen as starting points for new projects.

Packages in the oacs-5-9 channel.
Packages in the oacs-5-10 channel.