Forum OpenACS Q&A: Response to Data migration

Collapse
6: Response to Data migration (response to 1)
Posted by Michael Bryzek on
Although I'm not currently using Open ACS, I am in the middle of a data migration from a fully-custom, legacy app to an ACS data model (in this case ACS Java...).

We approached data migration by first looking at existing tools. If you're lucky enough to do a straight column-to-column migration, there are plenty of tools out there to help you. One example for Oracle is the Oracle Migration Workbench. Unfortunately, I doubt these kinds of tools will be useful when migrating to the object-oriented data models we use today.

We divided our migration work into two steps:

  1. Export data from the legacy application to an intermediate XML format.
  2. Import data from the XML document using the API's available in our version of the ACS.
The use of XML as an intermediate representation of the data had some very real advantages for us. Namely,
  • We expect to do future data migrations. Next time, some of the work will already be done for us. I do not yet have practical experience showing what percentage of our migration code will be reusable, but definitely parts of the XML to ACS process will be.
  • Two of us could work independently on the migration process. One person needs to understand the legacy data in depth, creating an XML document to represent it. The other person never has to think about the legacy app - only about parsing the XML and calling the correct API's.
  • The import process takes full advantage of our existing API.
The only major drawbacks we've seen with using XML so far is speed and maintenance. It is much slower to migrate data in this way then in direct migrations (e.g. using oracle's sqlldr). In our particular case, we do not have a lot of data so we do not have to worry about the performance of the migration. The maintenance issue will come up next time we do a data migration - as API's evolve, the importer must also be updated. This can be mitigated to some extent if your importer is written generically, but it is yet another package that needs to be maintained.

What do people think about using XML to help with data migration? It would be very useful if we had an XML Schema for OACS and a corresponding utility which knew how to instantiate objects based on the data in the XML.