Forum OpenACS Q&A: Postgres troubles

Collapse
Posted by Robert Ezman on
I'm having difficulty restoring a database with posgresql....

Actually I've never actually been able to sucessfully restore a
database....

What follows is the error log.  Or at least where the retore broke
down (btw I had to put in a bunch of carriage returns so the log
wouldn't mush together into one big mess:

NOTICE:  CREATE TABLE/PRIMARY KEY will create implicit index
'dev_db_menu_items_pkey' for table 'dev_db_menu_items'

ERROR:  Relation 'users_special_quotas' does not exist
invalid command .

ERROR:  Relation 'survsimp_surveys' does not exist
invalid command .

ERROR:  parser: parse error at or near "1"
invalid command .

ERROR:  parser: parse error at or near "1"

invalid command N

invalid command N

invalid command N

invalid command N

invalid command .

ERROR:  parser: parse error at or near "1"

invalid command .

ERROR:  DefineIndex: relation "survsimp_surveys" not found

ERROR:  Relation 'users_special_quotas' does not exist

ERROR:  Relation 'survsimp_surveys' does not exist

ERROR:  Relation 'survsimp_surveys' does not exist

ERROR:  Relation 'survsimp_surveys' does not exist

The dump has the following lines:

CREATE TABLE "users_special_quotas" (

        "user_id" int4 NOT NULL,

        "max_quota" float8 NOT NULL,

        "modification_date" timestamp DEFAULT sysdate() NOT NULL,

        PRIMARY KEY ("user_id")

);

Unofortunately the log has the following lines aswell:

ERROR:  Function 'sysdate()' does not exist
        Unable to identify a function that satisfies the given
argument types
        You may need to add explicit typecasts

Possibly that's why the table was not created.....

My question is why?

I'm using version 7.0.2 of postgresql.  I'm guessing that people are
able to restore the database otherwise no one would recommend using it
at all.  Unfortunately my confidence in postgresql is a bit shaken.

Any Ideas....

Also how do I contact the developers of postgresql.  It's not that
easy it seems.  Too bad they're not running xAcs.

Collapse
Posted by Dan Wickstrom on
There is a bug in the way postgresql dumps out the db so that everything is not in the right order.  An easy way to fix it is to edit the dump file and move the sysdate function declaration to the top of the dump file.  You will also need to move the plpgsql function handler declaration to the top of the file, if you don't have it defined in your template database.  The function handler definition should come before the sysdate function declaration.  I think this problem is fixed in pg 7.1, so when it comes out you will no longer have this problem.
Collapse
Posted by Don Baccus on
PG 7.1 appears to have fixed it "good enough for OpenACS 3.2", at least.

In the interim, another trick to pull is to do a "psql -f postgres.sql" in the /www/doc/sql directory of your OpenACS installation before running the dump restore.  Then ignore any "duplicate function" type errors you get.

Collapse
Posted by Robert Ezman on
I tried both the suggestions above and they worked....I'm happily closer to being comfident in the system...

I still had the following slew of errors though....

psql:newest_bk:13067: ERROR:  select rules target entry 16 has different size from attribute m_address

psql:newest_bk:13070: ERROR:  select rules target entry 7 has different size from attribute read_permission_p

psql:newest_bk:13072: ERROR:  select rules target entry 8 has different size from attribute email_bouncing_p

psql:newest_bk:13074: ERROR:  select rules target entry 9 has different size from attribute approved_p

psql:newest_bk:13088: ERROR:  select rules target entry 4 has different size from attribute team_leader_p

psql:newest_bk:13090: ERROR:  select rules target entry 3 has different size from attribute on_which_table

psql:newest_bk:13092: ERROR:  select rules target entry 2 has different size from attribute salary

psql:newest_bk:13098: ERROR:  view rule for ec_subsubcategories_augmented must be named _RETec_subsubcategories_augmented

psql:newest_bk:13108: ERROR:  view rule for ec_gift_certificates_approved must be named _RETec_gift_certificates_approved

psql:newest_bk:13109: ERROR:  view rule for ec_gift_certificates_purchased must be named _RETec_gift_certificates_purchased

psql:newest_bk:13112: ERROR:  view rule for ec_fin_transactions_reportable must be named _RETec_fin_transactions_reportable

The question I guess is: are these something that I have to worry about or are these benign errors? (If anyone has a explanation for the cause of the error that would be great too)

Collapse
Posted by Don Baccus on
I've seen the first class of error before in my own dumps and they appear to be benign.  I've not seen the errors in ecommerce before, but then again I'm not using ecommerce.

I'd suggest updating to PG 7.0.3 and re-dumping.  You can use the same  database binary files, i.e. you don't need to dump and reload in order to upgrade to 7.0.3.

I'd be interested in knowing if PG 7.0.3 has fixes for these problems.

Collapse
Posted by Robert Ezman on
The version of postgres I have installed is the Redhat rpm that came with Redhat v7.0.  I was just about to install v 7.0.3 of postgres with the rpm supplied from the postgres site.  Gnorpm didn't seem to see that a previous version (that is v7.0.2) was even present on my machine.

Does anyone know about the compatability of the redhat and "official" postgres rpms.

Has anyone done this upgrade?

(by the way thanks to all the answers to my slew of questions.)

Collapse
Posted by Don Baccus on
You know, I just had that problem with another, totally unrelated RPM on my RH 6.2 distribution.  I'd installed it, then decided to uninstall it.  This is now deprecated and it says "use -e".  "-e" said  the package wasn't installed. "-i" said the package was already installed.

Seems like a contradiction to me!

So the problem might not be with the RPM on the Postgres site...

Lamar Owen, who builds these RPMs for RH and the PG site, hangs around  here at OpenACS.  With a little luck he'll pop up and give you an answer.

Collapse
Posted by David Walker on
> "-e" said the package wasn't installed. "-i" said the package was
> already installed. 

Note that -e and -i use different syntax.
 rpm -e packagename
 rpm -i filename

the package name is usually something like "postgres" or
 "postgres-server" and the file name is something like
 "postgres-7.0.3-2.i386.rpm"
Waaaaay back in 2000, Don was wondering whether these
select rule's target entry 16 has a diferent size from attribute m_address
errors are still present in PG 7.0.3 pg_dumps. The last comment in this thread was nearly a year ago, so it appears that most people are beyond this.

However, I'm just moving a site to a new box and in the process am moving from PG 7.0.3 to PG 7.1.3-7, and I see these errors now -- in fact exactly the same ones others reported. Fortunately, otherwise there are no errors, thanks to the preload of postgres.sql (I presume).

This thread doesn't provide any definitive conclusion about whether these errors are benign, as you thought they were, Don. Any further ideas at this point? The db seems to work as intended (though I've got a different weird problem that I think is unrelated).

Is this something just to ignore?

Collapse
Posted by Don Baccus on
Boy, this was a long time ago!  I think it was related to some views that return null or something like that ...

Is there any reason you're not using PG 7.2?

Collapse
Posted by Stan Kaufman on
Is there any reason you're not using PG 7.2?

Main reason is that PG 7.1.3-7 is the current version for Debian woody (aka testing), the distro I use. PG 7.2 is still up in sid (aka unstable). Debian makes upgrading so easy that I hate to step off the One True Path into the custom compile route, though of course I'm happy to do that when it's important--such as with AOLServer. I gather that you're advising me to do so here, and I thus will likely do that.

However, the errors I was reporting for 7.0.3 pg_dump (and the topic of this thread) would occur if I were restoring that dump from 7.0.3 into a 7.2 DB, too, wouldn't they? My main concern was whether down the road a bit I'd find that the existing data might crap out unexpectedly, or whether things are OK.

Collapse
Posted by Don Baccus on
I'm not sure whether it would or wouldn't still fail with PG 7.2, actually.  Still ... if it were me I'd switch to PG 7.2.  IMO they - not Debian - get to decide which of there versions is stable and which are unstable :)
Collapse
Posted by Ken Kennedy on
The unstable/testing/stable monikers aren't indications of the stability of the program...more the newness of the Debian package itself. 7.2 will move into testing 2 weeks after it's been uploaded to unstable (barring any bugs in it or dependent packages). Shouldn't be long now. Or, if one is brave *grin*, you could just set your apt preferences to include unstable at a lower priority than testing, and then "apt-get -t unstable postgresql". That'll get postgresql and any necessary dependencies out of unstable directly, w/o having to move your whole distribution up there. I do that on my desktop box with the KDE2 packages, for example, to keep up to date.
Collapse
Posted by Stan Kaufman on
The unstable/testing/stable monikers aren't indications of the stability of the program...more the newness of the Debian package itself.
That's why I like to think of the releases in terms of their Toy Story names; they represent collections of stuff that are in the same cohort. Very important re dependencies etc. Things don't break so much. Terrific system, IMHO.
Collapse
Posted by Don Baccus on
I know that Debian has their own independent rating system, if you will.  It's just that I trust the PG group inherently.

PG releases involve a deep testing cycle among serious users of the software.  The Debian testing cycle ... well, PG isn't exactly the "fave rave" in the Linux world, and I would be thoroughly amazed if the Debian test cycle for PG was even 10% as relevant as the PG test cycle which involves folks with terra-byte scale live installations.

Collapse
Posted by Ken Kennedy on
I know that Debian has their own independent rating system, if you will. It's just that I trust the PG group inherently.

So do Debian-types, Don! It's NOT a PG specific thing. It's a Debian thing.

Part of the HUGE bonus of a Debian system is the excellent integration of packages into an entire distribution. Debian has a set of standards for how things work, how daemons are started, checks for dependencies on other packages, daemon running privs, file locations, menuing system for passing parameters to the app setup, etc. Part of the responsibilities of a Debian package maintainer making sure that the app follows "policy" on a Debian system. Nobody's perfect, so ALL packages go through the same process; they're uploaded to unstable, and banged on by all Debian users willing to do so. Bugs are reported...they may be actual application bugs, or they may be Debian package bugs. Once a package has been in testing for a certain amount of time and passes certain criteria, it automatically get moved into the testing distribution. Testing (presently known as "woody"; all releases have Toy Story-related names) is banged on HARD...and eventually becomes the next stable release.

Sorry for the long, off-topic ramble...but the Debian policy and dependency system kicks SERIOUS butt. The vast majority of the time, installing something like postresql on your system, starting from zero, goes something like:

abelard:~# apt-get install postgresql
Reading Package Lists... Done
Building Dependency Tree... Done
The following extra packages will be installed:
  postgresql-client 
The following NEW packages will be installed:
  postgresql postgresql-client 
0 packages upgraded, 2 newly installed, 0 to remove and 1  not upgraded.
Need to get 1721kB of archives. After unpacking 4276kB will be used.
Do you want to continue? [Y/n] 
Get:1 http://non-us.debian.org testing/non-US/main postgresql-client 7.1.3-7 [268kB]
Get:2 http://non-us.debian.org testing/non-US/main postgresql 7.1.3-7 [1452kB]
Fetched 1721kB in 18s (94.0kB/s)                                                           
Selecting previously deselected package postgresql-client.
(Reading database ... 39890 files and directories currently installed.)
Unpacking postgresql-client (from .../postgresql-client_7.1.3-7_i386.deb) ...

Old binaries should already have been saved.
Selecting previously deselected package postgresql.
Unpacking postgresql (from .../postgresql_7.1.3-7_i386.deb) ...

Found an existing database directory at /var/lib/postgres/data

Setting up postgresql-client (7.1.3-7) ...

The file /etc/postgresql/postgresql.env provides the normal set-up for
an ordinary user running PostgreSQL. It is automatically read by the
wrapper script for PostgreSQL user commands in postgresql-client.


Setting up postgresql (7.1.3-7) ...
Installing new version of config file /etc/init.d/postgresql ...
Installing new version of config file /etc/postgresql/pg_hba.conf ...
Installing new version of config file /etc/postgresql/pg_ident.conf ...
Installing new version of config file /etc/cron.d/postgresql ...
The format of PostgreSQL's configuration files has changed
radically. /etc/postgresql/postmaster.init and /etc/postgresql/pg_options
are no longer used.
* I had an old PG install...*
Do you want me to move your old configuration to the new files? [Y/n]Y 
Restarting PostgreSQL database: postmaster
No /usr/lib/postgresql/bin/postmaster found running; none killed.
The database is in an older format that cannot be read by
version 7.1 of PostgreSQL.

The postinstallation script should attempt to upgrade the database
automatically.  If it fails, it must be done by hand.
*** READ /usr/share/doc/postgresql/README.Debian.migration.gz FIRST! ***

The version 7.1 postmaster cannot be started until
this is done.
.

The postmaster has not restarted.

postgresql has found a pre-existing database, and believes that it will be
able to update it automatically to the new version.  There is some risk to
your data in this.  If you have not got an up-to-date backup of
/var/lib/postgres/data, you should say no at this point.

In the course of the update, your existing /var/lib/postgres/data
will be moved to a parallel directory and a new directory created in
its place.  The saved dump of your existing database will then be loaded
into the new database.  If you have a lot of data, this may take a
considerable time; if your database is sufficiently large, it may fail
for lack of disk space.

The backup of /var/lib/postgres/data will be on the same filesystem;
the dump file will be in /usr/lib/postgresql/dumpall/.

Do you want me to continue with the automatic update? [y/N] y
*dump...blah...*
This is the ASCII output of the dump for you to check:

-- postgresql-dump on Wed Mar  6 10:37:50 EST 2002 from version 6.5
connect template1
select datdba into table tmp_pg_shadow       from pg_database where datname = 'template1';
delete from pg_shadow where usesysid <> tmp_pg_shadow.datdba;
drop table tmp_pg_shadow;
copy pg_shadow from stdin;
kkennedy        1000    t       f       t       t       N      N
.
connect template1 kkennedy
create database mydb;
connect mydb kkennedy
connect - postgres
CREATE FUNCTION "plpgsql_call_handler" ( ) RETURNS opaque AS '/usr/lib/postgresql/lib/plpgsql.so' LANGUAGE 'C';
CREATE TRUSTED PROCEDURAL LANGUAGE 'plpgsql' HANDLER "plpgsql_call_handler" LANCOMPILER 'PL/pgSQL';
-- postgresql-dump completed on Wed Mar  6 10:37:55 EST 2002
On the basis of this dump, is it OK to delete the old database? [y/n] y
* I don't care...just an example...*

Destroying old database...
Copying /var/lib/postgres/data to /var/lib/postgres/data.6.5.3-26.save
Deleting /var/lib/postgres/data
Creating the new database structure
This database system will be initialized with username "postgres".
This user will own all the data files and must also own the server process.

Fixing permissions on existing directory /var/lib/postgres/data
Creating directory /var/lib/postgres/data/base
Creating directory /var/lib/postgres/data/global
Creating directory /var/lib/postgres/data/pg_xlog
Creating template1 database in /var/lib/postgres/data/base/1
DEBUG:  database system was shut down at 2002-03-06 10:38:59 EST
DEBUG:  CheckPoint record at (0, 8)
DEBUG:  Redo record at (0, 8); Undo record at (0, 8); Shutdown TRUE
DEBUG:  NextTransactionId: 514; NextOid: 16384
DEBUG:  database system is in production state
Creating global relations in /var/lib/postgres/data/global
DEBUG:  database system was shut down at 2002-03-06 10:39:12 EST
DEBUG:  CheckPoint record at (0, 108)
DEBUG:  Redo record at (0, 108); Undo record at (0, 0); Shutdown TRUE
DEBUG:  NextTransactionId: 514; NextOid: 17199
DEBUG:  database system is in production state
Initializing pg_shadow.
Enabling unlimited row width for system tables.
Creating system views.

Loading pg_description.
Setting lastsysoid.
Vacuuming database.
Copying template1 to template0.

Success. You can now start the database server using:

    /usr/lib/postgresql/bin/postmaster -D /var/lib/postgres/data
or
    /usr/lib/postgresql/bin/pg_ctl -D /var/lib/postgres/data -l logfile start

Checking that the postmaster is running
Starting the postmaster
Starting PostgreSQL postmaster
/usr/lib/postgresql/bin/postmaster -i -b /usr/lib/postgresql/bin/postgres -D /var/lib/postgres/data >/var/log/postgres.log 2>&1 &
Loading the database from /usr/lib/postgresql/dumpall/db.out 
Reloading databases...
You are now connected to database template1.
select datdba into table tmp_pg_shadow       from pg_database where datname = 'template1';
SELECT
delete from pg_shadow where usesysid <> tmp_pg_shadow.datdba;
DELETE 0
drop table tmp_pg_shadow;
DROP
copy pg_shadow from stdin;
You are now connected to database template1 as user kkennedy.
create database mydb;
CREATE DATABASE
You are now connected to database mydb as user kkennedy.
You are now connected as new user postgres.
CREATE FUNCTION "plpgsql_call_handler" ( ) RETURNS opaque AS '/usr/lib/postgresql/lib/plpgsql.so' LANGUAGE 'C';
CREATE
CREATE TRUSTED PROCEDURAL LANGUAGE 'plpgsql' HANDLER "plpgsql_call_handler" LANCOMPILER 'PL/pgSQL';
CREATE
Reload succeeded
Restarting PostgreSQL database: postmaster
Stopped /usr/lib/postgresql/bin/postmaster (pid 10202).
Starting PostgreSQL postmaster.
postmaster successfully started

Enabling the PL procedural language in all PostgreSQL databases...
peer validation request: pg user=postgres, peer user = postgres, auth_arg = sameuser
peer validation request: pg user=postgres, peer user = postgres, auth_arg = sameuser
plpgsql is already enabled in mydb
peer validation request: pg user=postgres, peer user = postgres, auth_arg = sameuser
peer validation request: pg user=postgres, peer user = postgres, auth_arg = sameuser
peer validation request: pg user=postgres, peer user = postgres, auth_arg = sameuser
peer validation request: pg user=postgres, peer user = postgres, auth_arg = sameuser
peer validation request: pg user=postgres, peer user = postgres, auth_arg = sameuser
plpgsql added to template1
Done. And sorry for the long post...I forgot I had the old v.6 db there from when I ran PG on this machine. I thought it'd be interesting to see how the automated dump and replace went.