Forum OpenACS Development: Comments on the new postal-address module (and kin)

Alfred, Thanks for the comments. The changes for address and phone will only map to HR-XML and people can add packages as they see fit using these as a base. There will also be a slight change to the data model in 5.0 (easily backported to 4.6.x)

Telecom-number and postal-address are services. They are not packages. They are analogous to the reference-* packages. People are free to use the underlying data model as they see fit.

Everything you describe can be accomodated with the HR-XML framework. The reason other packages use seperate data is historical; The original address book was almost useless beyond a personal list and was most definitly hardly internationalized. Plus, many package designers just added their own tables without regard for what was already in the toolkit (states and countries come to mind).

My contacts-lite package is just that. It was (like much of the code here) written for a client who was kind enough to let me release it to the OACS community. It does however show how to use the underlying services (although certainly not everything has been completed, I have other obligations and they don't involve these packages anymore except for my personal use).

As anyone who has followed my posts on reference data and contacts in general knows, I am very concerned with internationalized contact information. Modelling these packages after HR-XML fulfills this need.

I also saw a thread recently from someone saying they are going to implement a contact system and ignore any other tables in the schema (sigh).

I'm that person - and no i did not say i would not use postal-address just that i have worked without it until now and will integrate it later - when postal address is "stable". Things like the ones you are mentioning are the types of issues that kept me from designing it with postal address built in - but i need to street it will be EASY to add it later... once things like this are sorted out.

Your concern about the necessity of persons related to postal-addresses is one of mine as well that i would have brought up later. I want a "contact" to also be assigned an address and they won't necessarily be a person in the system... so i agree with Alfred on this one. Having a seperate mapping table, where you map and address_id to a object_id would take care of this - because presumeably all services that would use postal-address would have objects that the address can be linked to. I also don't think it is necessary for a postal-address to even be an acs_object. We won't be needing to do permissions on individual addresses and there is no need (in my view) to clutter up those other systems. A simple unique address id created by a sequence should be enough.

Now, regarding alfreds points

1. Fantasitic idea - i modeled mine after "deleted on this date" but yours is better.
2. I agree with Jon that HR-XML takes care of htis problem - its a matter of correctly inputting data into the HR-XML standard - and this requires the appropriate fields to be filled out for US costomers - but it also allows people in other countries to satisfy their address requirements. I also based my contacts addresses on HR-XML before i knew postal-address existed as a package and independenly came up to the same conclusion as Jon on this one so i'll say i think its the right one.
3. I think it makes more sense to store longitude/latitude with zip codes and reference it that way. or, do you know of reference data that says house number "1234" on street "X Avenue" is at thisxthat degrees? This is primarily a matter of having longitude and latitude data for every house (and we can't expect users to enter it correctly themselves).
4. If postal-address is a service then this suggestion should be handled by some other package that references the postal-address service.
5. Same point as above
6. With the contacts system i am creating this will be possible - though i don't think it should be mandatory throughout openacs, but be something that can be activated via a parameter. Many online services don't care about a persons address.

that should be "i need to state" not street - these addresses are getting me confused :)

Having a ref-address package does nothing. Address data is not reference data. (reference data is seldom changed and usually from a 3rd party.).

Unless someone can come up with a very compelling reason not to have addresses and telephone numbers acs_objects they are going to stay that way. Comparisons to the way "AD" did it aren't reasons.

The current model is designed to make all addresses and phone numbers objects for many reasons, including site wide search and auditability.

As Alfred suggested, I am planning on tcl procs to do address verification. This is not trivial though and the system now works the way it is. Also, none of this has any effect on the data model or HR-XML.

Everything you guys are talking about should be built on top of the telecom-number and postal-address services. They are only repositories of data, what you do with them is another package.

Marketing data has to be stored seperatly, that is another package.

Longitude/Latitude is already incorporated into the ref-zipcodes service. Please note that getting this data is country specific with many if not most countries not having it. It really has no place embedded in the core data model.

There will be some changes to persons:

alter table persons add column formatted_name varchar(200);
alter table persons add column preferred_given_name varchar(100);
alter table persons add column middle_name varchar(100);
-- need to drop the not null constraint, I believe PG 7.3 allows this.

Also, person__new et al need to be replaced. I already did this for PG and Oracle will be just as simple.

Having a ref-address package does nothing. Address data is not reference data. (reference data is seldom changed and usually from a 3rd party.).

That's what list brokers do ...

Jon, i forgot about site wide search - yea, i agree they should be objects... I agree that address verification via tcl is important - do you also intend to make an address "widget" that can be inserted as an element in forms? i think this would be a great thing - i.e. similar to the way date has a number of fields as an element i think address should also have a numbmer of them - possibly selected via a parameter via acs-templating that lets you select the default countries setup of that input field.

But Jon, as a repository of data, postal-address should not rely on persons - it should rely on itself. I really don't like the constraint on persons and in fact it would make it impossible for me to use it with my contacts app without creating bogus "persons" in all i'm suggesting is changing this part of the postal-adderss table creation script

    party_id            integer
                        constraint postal_owner_id_fk
                            references parties (party_id),

I would like it to become

    object_id           integer
                        constraint postal_owner_id_fk
                            references acs_objects (object_id) 
                        constraint postal_owner_id_nn not null,

since this field can be null in its current state - which i hadn't realize it was - i can use it. in fact i am going to use it and implement it today so i don't start doing things that aren't standardize. However, if that field is not entered then there could be no way of "tracking down" addresses "belong" i.e. you could track the address object 1234 by the use of a map table with contact object 4321. But if lots of apps end up depending on postal-address (which they hopefully will), then i think it would be good to be able to search the postal-address table for all addresses belonging to the objects that particular application owns...

Alfred - john's point that ref-addresses isn't a good idea is a very valid one - that's not the way it would work with openacs standards. we could have a ref-addresses package that stores all this "tiger" data you are talking about but it wouldn't make sense to put unique addresses in there. Do you know of any gnu or gpled reference data that would let us calculate longitude and latitude? As Alfred suggested, I am planning on tcl procs to do address verification. This is not trivial though and the system now works the way it is. Also, none of this has any effect on the data model or HR-XML. Everything you guys are talking about should be built on top of the telecom-number and postal-address services. They are only repositories of data, what you do with them is another package. Marketing data has to be stored seperatly, that is another package. Longitude/Latitude is already incorporated into the ref-zipcodes service. Please note that getting this data is country specific with many if not most countries not having it. It really has no place embedded in the core data model. There will be some changes to persons: alter table persons add column formatted_name varchar(200); alter table persons add column preferred_given_name varchar(100); alter table persons add column middle_name varchar(100); -- need to drop the not null constraint, I believe PG 7.3 allows this. Also, person__new et al need to be replaced. I already did this for PG and Oracle will be just as simple.

The constraint is not on persons it is on party and that is a whole different thing having nothing to do with a person. It could be any data (i.e. organization, group etc)

And parties is going to change in 5.0 anyway to normalize the tables correctly. (i.e. a party doesn't need an email). Part of this is to work with other methods of registration etc. See other threads for this discussion.

Here's a page on a 'Data Quality' service provider that's worth a look:

http://www.melissadata.com/Lookups/addressverify.asp

If you enter an address, you will get the following - the ones with my * in front - I think are useful in the users screen to profile your user base:

Address
*Area Code
Carrier Route - DPC
*Address Type
ZIP Code Type
*County (FIPS Code)
*Time Zone (Local time)
*Consolidated Metro Area (CMSA)
*Primary Metro Area (PMSA)
Representative, Party & District
  # (useful for a political site maybe)
Delivery Post Office
Latitude
Longitude
Census Tract

CMSA is VERY broad - I live half way up Connecticut and am in the Bridgeport PMSA and the NEW YORK CMSA.

Address and carrier route are for snail mail support - and there is more stuff you can buy.

My point being - I have this table of data in OACS - stuff users type in and provide. At some point the guy in marketing says - how many of our users are on the east coast? I say - well if you let me send our data out for geocoding I can let you know. From that point on you send incremental files once a month or whatever, but you always can determine how many users are in what regions. Time Zone as I said, is useful if you decide to call the person.

Latitude and Longitude are good if you are going to use any mapping applications.

The point being - I am hoping to have the "address" information *as it is currently viewed by data companies* stored in an isolated area - that way its a lot easier to do what companies actually do - send it out to get checked and coded, without worrying about the relation integrity issues of all the other tables it touches.

I think that what Jon and Matthew are proposing will support that - I just want to make you aware that this additional information should NOT just be discounted as "marketing" stuff - it is part of the physical reality of the address in question, and it makes it easier for marketing to accurately identify the user population of their site - where do they live? What city? What time zone? How many are within 50/500/5000 miles of me or any of my locations?

Nobody that I know of provides this info en masse for free, GPL or whatever, but even the non-profits I work with have enough budget to save themselves money on mailings to pay for the geocoding and address verification.
It's current best practice, it pays for itself, and I think its a mistake to leave it out of the primary location we intend to store addresses :)

Jon, Jon, Jon :)

Challenging me to find the information about Cuba is meaningless. As far as I know, it's a federal crime to conduct business there - and Amnesty International isn't permitted to visit the country - so let's just put that red herring aside, shall we? (http://web.amnesty.org/report2003/cub-summary-eng)
I have a feeling that my clients would be satisfied if I could tell them how many subscribers were from Cuba. Does finding one example like that actually discredit my point about all the other locations? Are you really worried about the HR-XML implications for Cuba?

I live in Connecticut - most of my users and clients are in the United States.

You live in Las Vegas - most of your users and clients are probably in the United States.

Matthew's address is at Berkeley - most of his users and clients are probably in the United States.

Those of my clients who conduct business on the Internet internationally are primarily concerned with Europe and Japan, Mexico and Brazil ..

As far as I have been able to tell, all of the mentioned countries are following this trend toward address quality - I already provided examples in US/UK/Germany - there is a page on the japanpost website but I'm not THAT interested .. If I run the address I showed above in Germany - that puts the site in Bayern as the Ort - same as the PMSA. The same kind of data is being provided all over the place.

My only point here has been and continues to be - an address is at a physical point in space. Those physical points in space have specific characteristics that rarely change - what major city are they near? What is their exact latitude and longitude? What time zone are they in?

I see them as all being intrinsically tied together - the only thing that DOES change is who is at the address. The melissadata example I provided doesn't say anything about who is there - it just gives you the 'fact table' about that location. The one thing that changes in that example is the political data which I agree is not relevant.

I'm gonna claim Godwin's Law here - I'm out :) If I see some package (or is it a service?) called postal-address-verification-and-spatial-constructs I will contribute what I can... In the meantime, I'll just send my files to Melissa Data and get good addresses.

The fact that you think my Amnesty International response to the Cuba non sequitur was actually anything related to Greenpeace is an indication you haven't been reading what I said in much depth - which is your right of course.

I don't really care about the address verification, which is what you and Matthew both seem to think I'm primarily suggesting. I even tried to spell out that doing address verification does NOT ensure valid addresses - the best you can do in a TCL proc or javascript is make sure it's "well formed" - just like an email address or a credit card number.

My two primary issues were
#1 - decouple addresses from people.
#2 - store appropriate information with the addresses - lat/lon, DMA, timezone, etc.

I appreciate your adventures in Cuba - I am working with a school in Ecuador to put up a dotLrn site. I think all the stuff we're talking about here is largely irrelevant to either of those cases. The corporate guys are the ones who like the marketroid stuff - this is a great sales tool in that market and it provides an additional revenue stream for the developer/hoster if they periodically ship the addresses out to be cleaned up.

I clearly got lost in the distinction between services and packages (is a service something that uses a service contract? or just something with no ADP pages?) Parties - the only references I found were from AD people in the party::permission_p -

"Wrapper for ad_permission to allow us to bypass having to specify the read privilege " sounds like a way to skirt the permission system?

and you said they are changing in 5.0? I also am a bit vague on the _rels mentioned in the thread here ..

I guess there are enough internals here that I'm not familiar with and can't find documentation on that I'll just shut up and see what comes out. It's likely that I'll just add the fields I want on your table and keep a private version.

Thanks for the lively dialog :)

I actually did read it, but also had some greenpeace code in an ssh window. I don't have any need for Amnesty Intl. either. But that is irrelevant. And, I also don't hold animosity towards them, I just don't have a use for their politics.

A service is meant to be used by other packages and has nothing to do with service contracts. They in general don't have adp pages, but that is not the real distinction. The adp pages would be for admin purposes. Also, services should only be mounted once (usually)

I also think you are confusing my proposal for a validation package. Your package can use a tcl proc to consolidate/validate do whatever you want to the data. If you want to send it out to xxx-data service, no problem, ecommerce does a similar thing with payment gateways. There used to be one payment method. It has since been rewritten to use service contract to extend the ecommerce package without touching the core data model.

A similar approach should be used in the address-validation package. I am sure that every list cleaning/validation service has different requirements so you write a gateway and then add specifics for each company. By trying to put the US centric address information in the core of postal-address, you are locking everyone into your preferred company.If there was a standard for data cleaning/validation that would be a different matter, but as far as I know that doesn't exist.

The changes in 5.0 (which certainly are only being talked about) are changes to parties to eliminate persons information at the party level.

A party is an entity that relates to an object. That can be a person, group, organization or even another object. A person is a specific type of party and once a person becomes registered they become a user. But, all of the types inherit from parties.

Addresses don't have to be attached to a person, they are only attached to a party. You don't even need that information to send an address for validation. Also, you can create a history of that address by simply creating a package that tracks address_id and party_id with dates or whatever you want, but this isn't part of postal-addresses.

Ironically, I just found this:

https://openacs.org/bugtracker/openacs/bug?bug%5fnumber=241

in the bugtracker - requesting incorporating a GPL-ed copy of the TIGER street address to lat/lon data :)

http://ofb.net/~egnor/google.html

I will make it a ref-tiger package. Give me a few days.
But, I believe most of it is already in ref-zipcodes.