Forum OpenACS Q&A: HOWTO - simple script with some tcllib packages and namespaces

I have been playing with various csv import methods and decided to stick to the 'standard' tcllib packages of csv and struct::matrix because they are fairly smart and good enough for what I need. I ran into two obstacles, so here' what I did for any others who may follow. After mucking around trying to get the standard tcllib to work, I just ran [ns_info tcllib] and decided to copy the csv.tcl and matrix.tcl files into that tcllib area. At that point my package requires worked fine. CSV was totally fine, there was a subtle behaviour on the struct::matrix library - it works like tdom does - you don't create a variable to hold the data, but rather, a proc that has various methods of munging its contents.

By default you call struct::matrix newMatrixName and create a matrix proc. Unfortunately it gets tangled in the current namespace, which is template:: in most OpenACS pages. I couldn't say struct::matrix somenamespace::newMatrixName - the method below inside the namespace eval is what ended up working properly across multiple invocations. Note also that it doesn't disappear. You can either newMatrixName destroy at the end of the script or do so the next time through as I have done below.

What is omitted below is the ad_return_template and various other "regular" Openacs page things ..

Having said that - the struct library shows great promise!

package require csv
package require struct

set str6 {666,"123,521.2",001,"Mary says ""Hello, I am Mary"""}
set str7 {777,"123,521.2",001,"Mary says ""Hello, I am Mary"""}
catch {
foobar::someCSV destroy
}
namespace eval ::foobar:: {
        struct::matrix ::foobar::someCSV
    #struct::matrix someCSV  -- will work also
}
csv::split2matrix foobar::someCSV $str6 "," auto
csv::split2matrix foobar::someCSV $str7 "," auto
set numcols [foobar::someCSV columns]
set numrows [foobar::someCSV rows]
set count "$numcols and $numrows"

set output [foobar::someCSV format 2string ]

Alfred,

have you tried ::struct::matrix to avoid intepretation within the current namespace?

/Bart

Bart - that hadn't occured to me. If you'll bear with the analogy - it feels like asking a very very dense valet for your car. Once you have provided every tedious detail of the description of the car, including which parking lot its in - if you're persistent you will get your car back.

I guess in reality the valets (struct::matrix) are lazy and will look in the current parking lot where you are standing unless you kick them in the double colon and make them work a little harder.

Alfred, just to get back to your real issue - depending on how well you need to handle errors, you can really easily suck csv files straight into a postgres table (or temp table) using the sql COPY command. From memory commas in a value have to be backslash escaped and it doesn't doo anything smart with quotes (although you can deal with that later using a simplke regex).

I would be tempted to write a simple perl tool to do the work (using one of the many CSV modules on CPAN) and just exec that...

That's good to know - I had hoped there was some psql csv import function.

My real goal though, is to write a package that allows an end user to upload formatted data - I envision csv, non-nested xml, and fixed width files as my import types.

What I am developing will allow the end user to upload a file, the struct::matrix package has some nice tools to analyze column widths etc ...

The wizard will guide the user to table creation based on regex type guessing, etc. Char fields are going to be T-shirt sizes S/M/L/XL/XXL varchar(8) .. varchar(max), other types will map to their native types.

Upload specs will be recorded (like in ms-access imex spec) so users have a template for later uploads.

Final step is a wizard/form to map the data into our acs_objects/acs_datatypes framework.

User should be able to then restrict access, assign notifications, create search on specified fields, and other functions provided by our toolkit.

Rationale: Power to the end user. Clean up data soup. Reduce recurring programming tasks.

This project was the motivation for my TIP (52) about abstract system catalog.

Similar project - www.quickbase.com