Forum OpenACS Q&A: Re: HOWTO - simple script with some tcllib packages and namespaces

Alfred, just to get back to your real issue - depending on how well you need to handle errors, you can really easily suck csv files straight into a postgres table (or temp table) using the sql COPY command. From memory commas in a value have to be backslash escaped and it doesn't doo anything smart with quotes (although you can deal with that later using a simplke regex).

I would be tempted to write a simple perl tool to do the work (using one of the many CSV modules on CPAN) and just exec that...

That's good to know - I had hoped there was some psql csv import function.

My real goal though, is to write a package that allows an end user to upload formatted data - I envision csv, non-nested xml, and fixed width files as my import types.

What I am developing will allow the end user to upload a file, the struct::matrix package has some nice tools to analyze column widths etc ...

The wizard will guide the user to table creation based on regex type guessing, etc. Char fields are going to be T-shirt sizes S/M/L/XL/XXL varchar(8) .. varchar(max), other types will map to their native types.

Upload specs will be recorded (like in ms-access imex spec) so users have a template for later uploads.

Final step is a wizard/form to map the data into our acs_objects/acs_datatypes framework.

User should be able to then restrict access, assign notifications, create search on specified fields, and other functions provided by our toolkit.

Rationale: Power to the end user. Clean up data soup. Reduce recurring programming tasks.

This project was the motivation for my TIP (52) about abstract system catalog.

Similar project - www.quickbase.com