Forum OpenACS Development: OpenACS 5.0 Upgrade Experiences

if ![apm_package_installed_p acs-lang] {

    apm_package_install -enable -mount_path acs-lang [acs_root_dir]/packages/acs-lang/acs-lang.info

}


if ![apm_package_installed_p acs-authentication] {

    apm_package_install -enable [acs_root_dir]/packages/acs-authentication/acs-authentication.info

}

I think you are correct. The package version isn't updated unless you install through the APM. I'll do a test and see how that works.

Yes, the user will have to upgrade those packages through the apm and uncheck the datamodel files.

This is only if you are upgrading from a version that did not support postgresql 7.3 to a versison that does support 7.3. Basically any pre-4.6.3 version to any version 4.6.3 or later.

Very bizarre.

Before running the upgrade script (on a new restore of the dump of course) could you see what happens if you try selecting from the view?

There are problems going to PG 7.2->PG 7.3 if you use the default limits on identifier length because the PG group bumped it from 32 to 64.  About time, but it means they can't guarantee a dump from 7.2 will reload correctly into 7.3.  But that's not the problem you're seeing (there are some threads for getting around that problem when you get that far).

Hmmm ... did you check the output of the dump?  Are you sure there were no errors when the view was created on the restore?  It sure looks like the underlying rule was created but the view itself was not.

Steps

1.Inspect at acs-admin/apm found acs-kernal to be version 4.6.2
2.made a dump of the database
3.Upgraded the code for acs-kernal to 4.6.3
4.upgraded the kernal from the apm
5.made a dump of the database
6.loaded the dump into pg 7.3 – I did not need to run the upgrade_helper script because my hosting provider (zill.net) has set the 7.2 character limit to 64 so it did not truncate the function names.
7.Sanity check the import by looking at number of users or objects
8.Made a copy of the code
9.Used this cvs command to get the code cvs -z3 -d :pserver:mailto:anonymous@openacs.org:/cvsroot co -r oacs-5-0 acs-core
My 5.0 code is now in the directory /home/aristoi/openacs-4
10.$cd openacs-4/packages/acs-kernel/sql/postgresql/
11.psql -f postgresql.sql – This generated a lot of already exist errors and some create functions
12.cd upgrade/
13. psql -f upgrade-4.6.3-4.6.4.sql
14.psql -f upgrade-4.6.4-4.6.5.sql
15.psql -f upgrade-4.6.5-4.6.6.sql
16.psql -f upgrade-4.6-4.6.1.sql  - This was a mistake I am already at 4.6.3
17.psql -f upgrade-4.7d-4.7.2d.sql
18.psql -f upgrade-4.7.2d-5.0d.sql
19.psql -f upgrade-5.0d-5.0d2.sql
20.psql -f upgrade-5.0d2-5.0d3.sql
21. psql -f upgrade-5.0d6-5.0d7.sql
22.psql -f upgrade-5.0d7-5.0d9.sql
psql:upgrade-5.0d7-5.0d9.sql:12: NOTICE:  rule _RETURN on view apm_file_info depends on table apm_package_files
psql:upgrade-5.0d7-5.0d9.sql:12: NOTICE:  view apm_file_info depends on rule _RETURN on view apm_file_info
psql:upgrade-5.0d7-5.0d9.sql:12: ERROR:  Cannot drop table apm_package_files because other objects depend on it
24)psql -f upgrade-5.0d11-5.0d12.sql
25)psql -f upgrade-5.0.0a4-5.0.0a5.sql
26)psql -f upgrade-5.0.0b1-5.0.0b2.sql
27)psql -f upgrade-5.0.0b2-5.0.0b3.sql
28)psql -f upgrade-5.0.0b3-5.0.0b4.sql
psql:upgrade-5.0.0b3-5.0.0b4.sql:788: ERROR:  Cannot change number of columns in view
psql:upgrade-5.0.0b3-5.0.0b4.sql:802: ERROR:  Cannot change number of columns in view
29) cd ~/openacs-4/packages/acs-service-contract/sql/postgresql/upgrade/
30)psql -f upgrade-4.7d2-4.7d3.sql
31)cd ~/openacs-4/packages/acs-authentication/sql/postgresq
32)psql -f acs-authentication-create.sql
33) cd ~/openacs-4/packages/acs-lang/sql/postgresql/
34) psql -f acs-lang-create.sql
BEGIN
psql:ad-locales.sql:44: NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index 'ad_locale_abbrev_pk' for table 'ad_locales'
psql:ad-locales.sql:44: NOTICE:  CREATE TABLE / UNIQUE will create implicit index 'ad_locale_name_unq' for table 'ad_locales'
psql:ad-locales.sql:44: ERROR:  Relation 'ad_locales' already exists
psql:ad-locales.sql:56: ERROR:  current transaction is aborted, queries ignored until end of transaction block
psql:ad-locales.sql:60: ERROR:  current transaction is aborted, queries
many more of theseLooking in my dump file I see that yes ad_locals does exist so I probably installed it at some point. I will try running the upgrade scripts.
33)cd upgrade/
34)psql -f upgrade-4.1-4.7d2.sql
35)psql -f upgrade-4.7d2-4.7d3.sql psql -f upgrade-4.7d2-4.7d3.sql
36)psql -f upgrade-4.7d3-4.7d4.sql
37)psql -f upgrade-4.7d4-4.7d5.sql
38)psql -f upgrade-4.7d6-4.7d7.sql
psql:upgrade-4.7d6-4.7d7.sql:3: ERROR:  Cannot drop table ad_locales because other objects depend on it
      Use DROP ... CASCADE to drop the dependent objects too
psql:upgrade-4.7d6-4.7d7.sql:28: NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index 'ad_locale_abbrev_pk' for table 'ad_locales'
psql:upgrade-4.7d6-4.7d7.sql:28: NOTICE:  CREATE TABLE / UNIQUE will create implicit index 'ad_locale_name_unq for table 'ad_locales'
psql:upgrade-4.7d6-4.7d7.sql:28: ERROR:  Relation 'ad_locales' already exists
psql:upgrade-4.7d6-4.7d7.sql:29: ERROR:  Cannot insert a duplicate key into unique index ad_locale_name_unq
DROP TABLE
39)psql -f upgrade-4.7d7-4.7d8.sql
Same errors as 4.7d6 to d7 above
40)psql -f upgrade-4.7d8-4.7d9.sql
41)psql -f upgrade-4.7d9-5.0d1.sql
42) created zzz-postload.tcl according to Dave;s instructions here. https://openacs.org/projects/openacs/5.0/upgrade-463-50

43)Added ns_param  tdom                ${bindir}/tdom.so to the ns_section "ns/server/${
44)Restarted server
45)error on trying to show a page
Error: tclop: invalid return code from filter proc 'invalid command name "parameter::get"': must be filter_ok, filter_return, or filter_break
Error durring load is
13/Dec/2003:22:56:11][18708.1024][-main-] Debug: PgBindCmd: sql =
      select locale
    from  ad_locales
        where  enabled_p = 't'

[13/Dec/2003:22:56:11][18708.1024][-main-] Error: Ns_PgExec: result status: 7 message: ERROR:  Attribute 'enabled_p' not found

[13/Dec/2003:22:56:11][18708.1024][-main-] Error: Error sourcing /home/aristoi/openacs-4/packages/acs-templating/tcl/acs-templating-init.tcl:
Database operation "select" failed (exception NSDB, "Query was not a statement returning rows.")
But when I try the query in psql it works fine.

Next problem is when I go to look at acs-admin it tries to log me in. From register/index e its doing the select_authority query with authority_id = NULL so the error is:

Query did not return any rows.
    while executing
"db_1row select_authority "
        select    [join $columns ",\n                  "]
        from      auth_authorities
        where      authorit..."

https://openacs.org/projects/openacs/5.0/upgrade-463-50

Where should these queries go?

Request Error
expected integer but got "MESSAGE"
    while executing
"string repeat $num_re $group"
    (procedure "lc_sepfmt" line 13)
    invoked from within
"lc_sepfmt $out $grouping $sep"
    (procedure "lc_numeric" line 20)
    invoked from within
"lc_numeric $num $fmt en_US"
    (procedure "lc_numeric" line 15)
    invoked from within
"lc_numeric $num_translated"
    invoked from within
"set num_translated_pretty [lc_numeric $num_translated]"
    ("uplevel" body line 10)
    invoked from within
"uplevel 1 $code_block "

Any ideas?

https://openacs.org/bugtracker/openacs/com/acs-kernel/bug?filter.status=closed&bug_number=517

Basic summary is that you don't seem to be able to manipulate named constraints on a db migrated from 7.2 which means a number of the upgrade scripts will fail.

The right answer would seem to be to put properly named constraints in place but that might be pretty hard to do. Fortunately I am leaving on vacation and can't help :)

These version numbers are not directly related to releases. For example, I'm not certain that the 4.6.3 release actually has an acs-kernel version 4.6.3, it might have been 4.6.4 or 4.6.5.

As for the 4.7 series, this was the original name HEAD, until we decided to rename it 5.0 instead, so yes, I'm pretty sure you need all of those as well.

Of course, thanks to branches, there's no obvious way to tell what the order should be for the 4.6.* and 4.7.* upgrade scripts, or whether they're all needed. We'll have to manually inspect each one of them. And cvs.openacs.org is currently down.

/Lars

cd
pg_dump kurup > mydb.dmp
dropdb dev-kurup
createdb dev-kurup
psql -f mydb.dmp dev-kurup
cd web/dev-kurup/packages/acs-kernel/sql/postgresql/upgrade/
psql dev-kurup < upgrade-4.6.4-4.6.5.sql 
psql dev-kurup < upgrade-4.6.5-4.6.6.sql 
psql dev-kurup < upgrade-4.7d-4.7.2d.sql 
psql dev-kurup < upgrade-4.7.2d-5.0d.sql 
psql dev-kurup < upgrade-5.0d-5.0d2.sql 
psql dev-kurup < upgrade-5.0d2-5.0d3.sql 
psql dev-kurup < upgrade-5.0d6-5.0d7.sql 
psql dev-kurup < upgrade-5.0d7-5.0d9.sql 
psql dev-kurup < upgrade-5.0d11-5.0d12.sql 
psql dev-kurup < upgrade-5.0.0a4-5.0.0a5.sql 
psql dev-kurup < upgrade-5.0.0b1-5.0.0b2.sql 
psql dev-kurup < upgrade-5.0.0b2-5.0.0b3.sql 
psql dev-kurup < upgrade-5.0.0b3-5.0.0b4.sql 
cd ../../../../acs-service-contract/sql/postgresql/upgrade/
psql dev-kurup < upgrade-4.7d2-4.7d3.sql 
cd ../../../../acs-authentication/sql/postgresql/
psql dev-kurup < acs-authentication-create.sql 
cd ../../../acs-lang/sql/postgresql/
psql dev-kurup < acs-lang-create.sql 
dev-kurup-start.sh

I just loaded up my before-upgrade DB and got this:

testkurup=# select * from persons where person_id=0;
 person_id | first_names  | last_name
-----------+--------------+-----------
         0 | Unregistered | Visitor
(1 row)
 
testkurup=# select * from parties where party_id=0;
 party_id | email | url
----------+-------+-----
        0 |       |
(1 row)

if {![apm_package_installed_p acs-lang]} {

    apm_package_install -enable -mount_path acs-lang [acs_root_dir]/packages/acs-lang/acs-lang.info
 
    lang::catalog::import -locales [list "en_US"]

}

-- kernel

@acs-kernel/sql/oracle/upgrade/upgrade-4.6.3-4.6.4.sql;
@acs-kernel/sql/oracle/upgrade/upgrade-4.6.4-4.6.5.sql;
@acs-kernel/sql/oracle/upgrade/upgrade-4.6.5-4.6.6.sql;
@acs-kernel/sql/oracle/upgrade/upgrade-4.7d-4.7.2d.sql;
@acs-kernel/sql/oracle/upgrade/upgrade-4.7.2d-5.0d.sql;
@acs-kernel/sql/oracle/upgrade/upgrade-5.0d-5.0d2.sql;
@acs-kernel/sql/oracle/upgrade/upgrade-5.0d2-5.0d3.sql;
@acs-kernel/sql/oracle/upgrade/upgrade-5.0d6-5.0d7.sql;
@acs-kernel/sql/oracle/upgrade/upgrade-5.0d7-5.0d9.sql;
@acs-kernel/sql/oracle/upgrade/upgrade-5.0d9-5.0d13.sql
@acs-kernel/sql/oracle/upgrade/upgrade-5.0.0b1-5.0.0b2.sql;
@acs-kernel/sql/oracle/upgrade/upgrade-5.0.0b2-5.0.0b3.sql;
@acs-kernel/sql/oracle/upgrade/upgrade-5.0.0b3-5.0.0b4.sql;

-- Service Contract

@acs-service-contract/sql/oracle/upgrade/upgrade-4.5-4.5.1.sql;
@acs-service-contract/sql/oracle/upgrade/upgrade-4.5.1-4.6.sql
@acs-service-contract/sql/oracle/upgrade/upgrade-4.7d2-4.7d3.sql;
@acs-service-contract/sql/oracle/upgrade/upgrade-4.7d3-5.0d1.sql;

-- Acs Authentication

@acs-authentication/sql/oracle/acs-authentication-create.sql

-- Acs Lang
@acs-lang/sql/oracle/acs-lang-create.sql

I put some time aside this weekend to do the 4.6.3 -> 5.0.0 upgrade on my peronal site. I got pretty far (note to anyone else atempting this: look at and use the script -> upgrade with documentation alone does not work at all).

Was able to get through a lot of errors, but not this one: When I try to login (step 8: If you can login, visit /acs-admin/apm and upgrade acs-kernel and acs-service-contract and uncheck the data model scripts) I get a request error and this is was the error.log says:

[21/Mar/2004:09:04:39][30040.5126][-conn2-] Error: GET http://blesius.org/register/?return%5furl=%2f
referred by "http://blesius.org/"
no value given for parameter "key" to "_"
    while executing
"_ acs-subsite.Log_In"
    invoked from within
"list [_ acs-subsite.Log_In] ok"
    invoked from within
"list [list [_ acs-subsite.Log_In] ok]"
    invoked from within
"set login_button [list [list [_ acs-subsite.Log_In] ok]]"
    ("uplevel" body line 77)
    invoked from within
"uplevel {
          # Present a login box
#
# Expects:
#   subsite_id - optional, defaults to nearest subsite
#   return_url - optional, defaults to Your..."
    (procedure "code::tcl::/web/crb/packages/acs-subsite/lib/login" line 2)
    invoked from within
"code::tcl::$__adp_stub"
    invoked from within
"if { [file exists $__adp_stub.tcl] } {

      # ensure that data source preparation procedure exists and is up-to-date
      adp_init tcl $__adp_stub
..."
    ("uplevel" body line 3)
    invoked from within
"uplevel {

    if { [file exists $__adp_stub.tcl] } {

      # ensure that data source preparation procedure exists and is up-to-date
      adp_init t..."
    (procedure "adp_prepare" line 2)
    invoked from within
"adp_prepare "
    (procedure "template::adp_parse" line 30)
    invoked from within
"template::adp_parse [template::util::url_to_file "/packages/acs-subsite/lib/login" "$__adp_stub"] [list return_url "${return_url}" no_frame_p "1" auth..."
    invoked from within
"append __adp_output [template::adp_parse [template::util::url_to_file "/packages/acs-subsite/lib/login" "$__adp_stub"] [list return_url "${return_url}..."
    ("uplevel" body line 17)
    invoked from within
"uplevel {
          set __adp_output ""

    set __adp_master [template::util::url_to_file "[ad_parameter -package_id [ad_conn subsite_id] DefaultMaster ..."
    (procedure "template::code::adp::/web/crb/packages/acs-subsite/www/regis..." line 2)
    invoked from within
"template::code::${template_extension}::$__adp_stub"
    (procedure "template::adp_parse" line 68)
    invoked from within
"template::adp_parse [file root [ad_conn file]] {}"
    (procedure "adp_parse_ad_conn_file" line 7)
    invoked from within
"$handler"
    ("uplevel" body line 2)
    invoked from within
"uplevel $code"
    invoked from within
"ad_try {
                $handler
            } ad_script_abort val {
                # do nothing
            }"
    invoked from within
"rp_serve_concrete_file [ad_conn file]"
    (procedure "rp_serve_abstract_file" line 60)
    invoked from within
"rp_serve_abstract_file "$root/$path""
    ("uplevel" body line 2)
    invoked from within
"uplevel $code"
    invoked from within
"ad_try {
        rp_serve_abstract_file "$root/$path"
        set tcl_url2file([ad_conn url]) [ad_conn file]
        set tcl_url2path_info([ad_conn url]) [ad_conn path_inf..."

Suggestions?

It is hard to visit /acs-lang/admin/load-catalog-files when I can not login.

I already tried what Robert mentioned above (loading /foo.tcl with lang::catalog::import -locales [list "en_US"] in it).

Just to make sure I removed the "-locales [list "en_US"]" part so it more closely resembles /acs-lang/admin/load-catalog-files and tried it again. No luck, same error on registration.

Thanks, but I aborted the upgrade and restored from backups.

Going to put the two postgres versions up on different ports in parallel so I can tool with it as I have time.

Here is an improved script from Jeff (for posterity)

rename _ original_
proc _ {{key {}} {vars {}}} {
  if {[string is space $key]} {
    return NULL_KEY
  }
    return [uplevel [list original_ $key $vars]]
}

Round two, coming soon to a forum near you.

[12/Apr/2004:00:57:24][29728.114696][-conn4-] Error: GET http://myserver.com/register/?return%5furl=%2facs%2dadmin%2f
referred by ""
no value given for parameter "key" to "_"
    while executing
"_ acs-subsite.Log_In"
    invoked from within
"list [_ acs-subsite.Log_In] ok"
    invoked from within
"list [list [_ acs-subsite.Log_In] ok]"
    invoked from within
"set login_button [list [list [_ acs-subsite.Log_In] ok]]"
    ("uplevel" body line 77)
    invoked from within
"uplevel {
    	  # Present a login box
#
# Expects:
#   subsite_id - optional, defaults to nearest subsite
#   return_url - optional, defaults to Your..."
    (procedure "code::tcl::service0/packages/acs-subsite/lib/..." line 2)
    invoked from within
"code::tcl::$__adp_stub"
    invoked from within
"if { [file exists $__adp_stub.tcl] } {

      # ensure that data source preparation procedure exists and is up-to-date
      adp_init tcl $__adp_stub
..."
    ("uplevel" body line 3)
    invoked from within
"uplevel {

/packages/acs-templating/tcl/0-procs.tcl
/packages/acs-lang/tcl/lang-init.tcl

error "Unterminated element" at position 672
"56"&
        <br />
        <input type=submit value="Search" name="t">
      </f <--Error-- orm>
    
<hr noshade>
    <address><a href="mailto:webmaster@openacs.org">"
    while executing
"dom parse -simple [lindex $args 1]"
    (procedure "xml_parse" line 4)
    invoked from within
"xml_parse -persist $manifest"
    (procedure "apm_get_package_repository" line 30)
    invoked from within
"apm_get_package_repository -repository_url $repository_url -array repository"
    ("uplevel" body line 30)
    invoked from within
"uplevel {
    	  ad_page_contract {
    Install from local file system
} {
    {package_type "apm_application"}
    {upgrade_p 0}
    {repository_url ..."
    (procedure "code::tcl::/var/www/aolserver4/lal.com/packages/..." line 2)
    invoked from within
"code::tcl::$__adp_stub"
    invoked from within
"if { [file exists $__adp_stub.tcl] } {

      # ensure that data source preparation procedure exists and is up-to-date
      adp_init tcl $__adp_stub
..."
    ("uplevel" body line 3)
    invoked from within
"uplevel {

    if { [file exists $__adp_stub.tcl] } {

      # ensure that data source preparation procedure exists and is up-to-date
      adp_init t..."
    (procedure "adp_prepare" line 2)
    invoked from within
"adp_prepare "
    (procedure "template::adp_parse" line 30)
    invoked from within
"template::adp_parse [file root [ad_conn file]] {}"
    (procedure "adp_parse_ad_conn_file" line 7)
    invoked from within
"$handler"
    ("uplevel" body line 2)
    invoked from within
"uplevel $code"
    invoked from within
"ad_try {
                $handler
            } ad_script_abort val {
                # do nothing
            }"
    invoked from within
"rp_serve_concrete_file [ad_conn file]"
    (procedure "rp_serve_abstract_file" line 60)
    invoked from within
"rp_serve_abstract_file "$root/$path""
    ("uplevel" body line 2)
    invoked from within
"uplevel $code"
    invoked from within
"ad_try {
	rp_serve_abstract_file "$root/$path"
	set tcl_url2file([ad_conn url]) [ad_conn file]
	set tcl_url2path_info([ad_conn url]) [ad_conn path_inf..."

What did I miss?

basically, somewhere in acs-lang, there was a drop that didn't cascade, which didn't allow en_US to be inserted.  Had to manually go through the upgrade scripts and find the two drops that didn't cascade, and paste in the creates.

I needed the acs-lang patch in zz-postload.tcl, and the zz-hack.tcl that is mentioned above.

The other gotcha was the fact that I hit page-down twice, and missed the 'upgrade via /acs-admin/apm/' text.

Also, I had to create a file with this in it to reimport the message catalogs since en_US didn't exist.

lang::catalog::import -locales "en_US"

Once Dave set me straight, it was pretty smooth.  All told, roughly 9 hours of actual time were spent.  Being an OpenACS novice, that's probably slow compared to someone comfortable with OpenACS.  I only had 6 packages and had no custom code on the 4.6.3 site.

After that upgrade, I cleaned up two 5.0-beta* and a 5.0.1 install, moved two from a chroot, eliminated AOLServer 3 on that machine and now have 5 sites running 5.0.4.

A direct comparision puts 5.0.4 noticably faster than the same site when it was running 4.6.3.

Changelog for 5.0.x:
https://openacs.org/doc/openacs-5-0-0/release-notes.html

Someone else had that problem as well. I posted on that thread, as did Lars. So you might be able to find it by searching our postings. Or actually, I think it was in bug-tracker.

We eventually added href back in to the set of allowed attributes. I didn't think that error was present in 5.0.4, but maybe it is...?

It works for viewing pages, but when you try to save an edit you make:

Request Error

Database operation "0or1row" failed (exception NSDB, "Query was not a statement returning rows.")

ERROR:  function etp__create_new_revision("unknown", "unknown", "unknown", "unknown") does not exist
HINT:  No function matches the given name and argument types. You may need to add explicit type casts.

SQL:
select etp__create_new_revision('1096', 'co2', '292', '7716');

I'm on Postgres 7.4, and still on OpenACS 5.0