Forum OpenACS Development: OpenACS Attribute Management - Requirements/Requests
Don, Dave and I had a discussion on IRC today about openacs attributes. Here are the issues we addressed, as far as I recall (I'm sure I'll be corrected where necessary). We agreed that attribute storage should use the core datamodel. This means that we will attempt to use the core where possible and the facilities available to us in the acs_attributes and acs_attribute_values tables, with modifications if needed. These tables do not address some of the functionality we are looking for in an attribute management system.
- Attribute type permissions - certain attribute types may
need to use the permissions system to control whether or not
a user should be allowed to view or edit a particular
attribute type. In order to accomplish this we have proposed
two solutions.
- One solution is to modify the acs_attributes table attribute_id field (this attribute id refers to an attribute type not an attribute value), and instead of having it simply be the primary key it could reference the acs_objects table where each acs_attribute type would have a associated acs_attribute object ( of the newly created attribute type acs_attribute ). This would mean that the acs_attributes table would behave much like other object tables. This approach would require that we make sure that we upgrade the attribute_id in all packages that may use the attribute_id as a key for something - we off the top of our heads couldn't think of places that used attribute_id other than acs_attribute_values, but we need to verify this. Upgrade would require changing the ID in all referenced tables, the acs_attributes table and then once all attribute_id's have been made into acs_objects.
- Another solution is to add a column attribute_object_id to the acs_attributes table. This would have to reference acs_objects, but it could have a null value. This means that attribute_id's are not upgraded. Then when we want to check permissions on an attribute_id we would need to get that ID and check permissions on it. Upgrading would be easy, but the acs_attributes table would not follow the format that most tables have in openacs - i.e. where the object_id is the primary key.
- Objects as Attribute Values - certain attributes are in fact objects. For example the home_address of a person object is a postal_address object itself. We need a way for objects to be attribute values of other objects. I.E. person 1234 has an attribute_type of home_address with a value of 4321; 4321 would be the postal_address object_id. Other uses would be binary files that are stored in the content repository could be references by their item_id and associated with other objects. Our storage solution needs to work for both type_specific and generic storage types. As far as I recall we decided that although acs_rels is the "logical" choice we would still need to store an object_id (i.e. the rel_id) to reference it so in order to increase performace we will stimply store the object_id as an attribute value. I'm still not sure how we would know what table to look in for the values associated with that object_id. I'm hoping Don or Dave can explain this further. Would we add another field to acs_attributes? The acs_attributes table stores what attributes the original object uses, not what the "sub object" is being referred to.
- ACS Widgets - we need to be able to associated widgets with attributes. This can be accomplished by creating an acs_widgets table with a primary key of widget_id, and adding a column to the acs_attributes table called widget_id, which must references the acs_widgets table, but may also have a null value (for those attributes that don't have associated widgets).
- Generic Storage of Multifaced Data - some data such as that resulting from richtext widgets has two associated values - the content and the mime type. Other data types such as binary data are not stored well in a text field - such as the one currently present on the generic acs_attribute_values table. We are leaning towards storing everything in text. We could possibly also create necessary compound data types for things such as richtext widgets that would need content and mime_type storage.
Concerns? Preferences? The scope of this discussion is only dealing with attribute storage and form creation. Object templates is a discussion for another day.
You can use the same technique as you did to retreive the information for the original object. That is, once you have the related object_id you can query the object type definition to find out how to get the attributes of the related object. I think there will be two ways to get the attributes. A Tcl proc that would support generic and specific stored attributes. And a view that would only be able to get type specific storage attributes similar to the ones now used in the content repository.
Re: generic storage of data, the resoning is that the existing system works, so we should work with the existing system as much as possible. If we later find we need to expand it to include storage for other datatypes, we can change it at that time, but this should reduce the initial complexity of the implementation. Don and I felt that generic storage of attributes would be more rare, and any instance where precision or greater performance is required would indiciate type specific storage of the attributes.
The (slightly opaque!) discussion yesterday seemed to be about 'how' - I really think we have to determine the 'what' first. In fact, most of the distracting diversions in the discussion seemed to be caused by lack of agreement over the objective!
Matthew Geddert and I have been working on a requirements document to provide some focus to the discussions. I have posted the first draft below.
It was unfortunate in a way that this was not circulated before the discussion yesterday because the requirements document deals with many of the issues that came up in the IRC discussion yesterday. In particular we have covered the issues of generic versus typed data, extent of the use of the CR object model, use of acs_rels, re-use of existing data model instead of creating new, revision management and logging and integration of the datamodel.
These were the ideas that Matthew and I were kicking around and need to be reviewed publically. Some of these were covered in the IRC discussion but no conclusive decision seemed to have been made. At least the good news is that we are all thinking along the same lines.
I propose that we step back for a moment or two. The summary of the IRC that Matthew has posted above will be useful in due course.
The steps that I propose are:
1) Agree and finalise the requirements doc
2) Submit the requirements doc for public review
3) Revise and publish requirements doc
4) Draft the datamodel solution (that is the stage at which we should have the discussions like yesterday's)
5) Document the data model and circulate
6) Revise/ optimise datamodel design and publish
Only then can we progress as a group with all energies in the one direction. Without this we all end up sparking off in different directions - and poor Matthew will get frustrated by having to change things half way through. If that happens nothing will ever be finished.
Don made the point that we should not be forcing people to have attributes as objects. In the requirements draft you will see that what we proposed a scheme that eliminates that issue.
I think that we should avoid pe-deciding the data model at this stage. Once we agree about the objectives we can prepare some proposals for the 'ideal' way to structure the data. We will have to address the major issues of performance and scaleability as part of this data model specification work - which ought to involve some actual testing of possible options. We will almost certainly have to look at ways to optimise the structures but the 'gold standard' option should be designed first and then optimised rather than trying to design AND optimise at the same time.
I hope that this is helpful. Requirements draft follows.
Regards
Richard
ACS Attribute Management System Requirements
Document written by Matthew Geddert and Richard Hamilton
I. Introduction
This is a requirements document for the OpenACS Attribute Management System (AMS). AMS provides an abstraction layer between the content repository, the templating system and client modules such that a developer can create and implement user extensible data structures.
II. Vision Statement
To make websites as easy as possible to learn to use, a consistent ‘look and feel’ of the user interface is important. For site publishers and administrators, separating aspects of the UI from the underlying program logic makes for easier maintenance and more efficient workflow. The OpenACS supports this by providing the ACS Templating system which separates elements of program logic from those of design. An important part of the ACS Templating service is the ‘ad_form’ procedure. This procedure provides a convenient declarative syntax for rendering data driven html ‘forms’ for information display and edit pages in web browsers, taking care of all ancillary aspects of form driven functionality including widget allocation, data validation and submission.
Similarly, to efficiently support central services such as security and revision management, and to provide abstraction from the RDBMS in the form of a content delivering API, the OpenACS stores information in a central repository known as the ACS Content Repository (CR). Developers are encouraged to use the content repository for new modules rather than implement their own application specific data models (which would then fall outside the capabilities of the general OpenACS content management applications).
The OpenACS CR data model is designed to be extremely flexible and supports the modelling of hierarchically organised entities (referred to as content items or objects) to which attributes can be attached in a manner analogous to attributes in a single table in an RDBMS. CR content objects can further be grouped into folders whereby all objects in the folder become descendents of the folder object (which is itself a special type of content object). To provide almost unrestricted flexibility, CR content items can also be related to each other in arbitrary ways through the acs_rels data structure.
This flexible data model provides for the possibility of creating applications that allow site users to exert direct control over attributes stored in the CR so that applications can be customised in extremely flexible ways without additional program logic being required. In order to realise this goal functionality is required that can create user pages for displaying and modifying data structures programmatically. To achieve this a means of storing information about dynamically created data structures must be designed such that calls to ad_form can be generated automatically following an AMS API call according to the structure of any specified data.
The AMS will provide:
- Mechanisms that allow application developers to more easily exploit the power, convenience and flexibility of the content repository without needing to have any understanding of its structure or API.
- Abstraction from the details of ad_form specifications.
- Abstraction from the db/CR layers for module level data retrieval.
- Mechanisms that allow authorised users to customise data structures and definitions in OpenACS applications.
A system for maintaining an audit trail at various levels of atomicity entirely at the control of the user upon data definition.
III. System Overview
The AMS will comprise:
· A data model that conforms to and/or integrates fully with and enhances the existing CR data model whilst eliminating the need for separate CR tables for each CR object type managed by AMS. (This will be possible since AMS calls will be the standard way to access AMS managed content objects).
· A library of tcl API calls which
a. support and simplify the use of the CR in application development.
b. return appropriate datasets from the content repository for display.
c. return appropriate ad_form input specifications for data manipulation.
d. expose attribute management functionality to the developer for inclusion in new modules.
e. provide developer access to the underlying AMS pl/pgsql functions.
f. provide an admin user interface that developers can include in their modules where users can manage attributes related thereto.
IV. Use-cases and User-scenarios
At present it is typical that application logic expects data in non-flexible form and requires query output in a predetermined form. Accordingly the SQL that provides data to tcl pages itself depends upon the fixed underlying data model. Inclusion in applications of additions to that data model require changes to both SQL and tcl code layers.
The AMS will be designed as an OpenACS service that provides attribute management services to modules for programmers to incorporate in their applications. As noted in the requirement for ACS Templating programmers are generally responsible for defining and implementing the application logic of a site. Just as ACS Templating separates presentation from application logic, so AMS will separate data definition from application logic.
AMS must provide mechanisms that support these requirements and allow the programmer to create ‘empty vessel’ applications that the user can customise for their own specific purposes.
Pages delivered by ACS Templating are split into two component parts. The presentation component is concerned with how to display the page, the underlying logic component contains the application logic that provides the page output data. It is in this logic component that calls to AMS will be used, in place of custom queries and data handling.
What is needed is:
- A notation and API for the programmer to generate the data to be displayed. In OpenACS, we call the data that we wish to display a ‘data source’ or ‘page property’.
Application logic is driven by the inputs the page gathers from the user request (e.g. the HTTP request), and the computation that the page must perform on this input. This computation will generally create and populate the data sources and page properties. Where in the past data sources tended to be connected to database queries, now (at least for user customisable attributes) they will be connected to an AMS API call.
- A mechanism for the developer to influence the specification that AMS provides to ad_form for the rendering of forms from customised data structures.
- A mechanism and API for the programmer to use when creating admin pages in his application that support user defined data structures.
- A notation and API for defining attributes and records where a record will be an OpenACS CR ‘record’ object that has child objects that are attributes (in this way AMS can support user defined complex compound data types analogous to the current OpenACS standard for ‘typed’ data in the CR).
Use-Case 1
Danny Dreamer wants to write a ‘contacts’ package where users can specify data structures according to their needs. He instantiates his new package and writes some admin pages that call the AMS API so that he can define a test data structure. He now constructs his user pages just as usual but using calls to AMS rather than to the database API or the CR direct. Because he wants to simply ‘attach’ arbitrary data to acs parties in his application Danny does not define an AMS record.
By doing this all of the detail of defining acs objects and creating CR object tables is abstracted from Danny in the writing of his application. In addition, because AMS is fully integrated with the CR, he will receive all the benefits in his application of his users’ customised data being part of the CR object system (i.e. permissions etc.) and a flexible and customisable logging and revision management scheme (at field level) which is at his users’ control. Also, because attributes associated with each package instance are children of a package ‘root’ folder, his new contacts package will be instance aware and permissions will be available by site node and ACS party.
Use-Case 2
Olly Optimist decides to write a new Edit-This-Page module for the OpenACS one weekend (he has to complete it by Monday because he is moving house and won’t have time to finish it after that!). He wants to have an admin interface that allows users to create custom page types and custom applications just as ETP now does. Unlike Danny he does not want a log of every change to every field, he simply wants to keep revisions of changes to a page (i.e. an AMS record).
He instantiates the original ETP application (minus the CR init scripts) as the basis for his new package and writes some admin pages that call the AMS API so that he can define a test application and a test record structure. He now finds that using AMS he can write the new ETP quite quickly, particularly given that the bulk of his admin pages are provided for him by AMS.
His application is finished quickly and works well because the difficult parts are already taken care of by the AMS service which is (by now hopefully) fully de-bugged. He shares all the benefits with Danny of the CR however unlike Danny’s application, Olly’s new ETP logs revisions at the ‘record’ (page) level.
Some months down the line Olly finds that one of his friends has thought of an application for the ETP package that requires him to log ALL changes at the field level. Olly looks anxiously at the API docs for AMS and discovers to his relief that this is easy to accomplish. All he has to do is define attributes as NOT children of an AMS record and make the appropriate logging selections using the AMS API data definition admin pages which he used to build his ETP application admin pages.
Olly’s friend is very impressed!
V. Related Links
· Design document (to be written)
VI.A Functional Requirements
· 10.0 A Common Solution
Programmers and users should only have to learn a single system that serves as a substrate for all the functionally specific modules in the toolkit.
10.0.1
The system should not make any assumptions about how pages should look or function. AMS usage should have no functional impact on design issues other than in so far as modules such as list builder and ad_form influence the creation of a standardised and consistent user interface across modules. This positive influence should be supported by AMS.
10.0.3
Users should be able to recognise the attribute customisation admin pages irrespective of the module that they are customising. Once a user has learned how to use the AMS attribute management pages they should be able to apply that knowledge throughout OpenACS without additional learning.
10.0.5
Programmers should be able to regard AMS as an abstraction of the CR such that generic content browsers should provide meaningful information about AMS managed data hierarchies. Programmers should also be able to regard AMS as an abstraction of ACS Templating (including specific support for ad_form, list builder, acs widgets and Jon Griffin’s new paginator).
10.5 Programmer's API
It must be easy to use/integrate the AMS with any application or to add it in. This implies a stable, simple, and comprehensive API for programmers to use when writing or converting modules.
10.5.1 Data Specification Publishing
AMS must be able to pass on information about data sources on demand so that logic code and presentation layers can be built to be adaptable.
10.5.2 Validation of User Specified Structures
AMS must be able to gather and deliver the data that the data specification promises. This means that if a user is permitted by AMS to change/add/delete an attribute then there must be no errors thrown when data is requested from that attribute.
10.5.3 Obsolescence of Data
AMS must be able to handle discontinued, deleted or obsolete data in accordance with the logging behaviour specified by the user at custom attribute creation time. AMS must handle intelligently the situation where the user wishes to change the logging behaviour of a data item.
10.5.4 Customised Data Structure Changes
AMS must be able to permit attributes that have not been associated with a record to be associated with a record at a later date (and vice versa). This should serve as a mechanism for altering revision behaviour, and as a means of grouping already stored attribute data together into useful effective ‘tables’. Where logging has already occurred at attribute level, this is to be discarded when the attribute is allocated to a record. Where an attribute is moved from a record to become independently revision logged, there will similarly be no historical information available.
10.5.5 Recreation of ‘points in time’
For independent attributes AMS will be able to provide the recorded ‘live’ attribute value for any point in time since attribute definition. Any time point outside the range of the logged values should return ‘undef’. For attributes grouped into AMS records the recorded ‘live’ revision of the entire record will be available.
· 11.0 Usability
Programmers should be able to develop default data structure specifications using their standard tools for writing and maintaining code on the server.
VI.B Non-functional Requirements
· 100.0 Distribution
It must be possible to release AMS both as part of the ACS and as a separate module. When distributed as part of the ACS, all documentation, examples, and source code must follow ACS standards. This includes but is not limited to using the db_api, ad_page_contract, list builder and ad_form appropriately.
· 110.0 Performance
The AMS must not cause any performance problems to a site. It must be fast and efficient, and it must not slow down page load speed by more than 10% versus a standard Content Repository attributes implementation.
VII. Revision History
Document Revision #
Action Taken, Notes
When?
By Whom?
0.1
Creation
22/11/04
Richard Hamilton
I like the goals and requirements you've outlined, and I think AMS will be a big step forward for OpenACS. Wish I had more feedback. :)
OpenACS has proven that it can be adpated. I really don't want to hold up development for years (again) waiting for the "perfect" design. We can get a huge amount of benefit by working on the most necessary features and adding on when we see a need.
So basically I think a must shorter list of very explicit requirements is all we need to work on this.
This is one of the places where I want to make sure we don't add complexity in order to provide a "complete" soltuion to a problem that does not exist.
Please also note that there IS a full tcl api for the content repostiory on HEAD so that is mostly already addressed.
Firstly to ease any anxiety :), I agree that we definitely do not want to introduce long delays. Certainly no-one is suggesting years! Actually this needs to be agreed in a matter of days and I know that Matthew needs to progress very quickly for his own requirements. And of course I agree that the simpler the solution the better.
Just to clarify, use case 2 is about ETP which discusses storing a revision as per the current ETP package. It is use case 1 that talks about storing revisions at attribute (i.e. field level) and this was introduced intially because Matthew has/had this as a specific requirement for his own needs.
For my part I am not clear why this particular issue has become so contentious since if all attribute 'values' are associated with a CR object such as 'ams_attribute' the revisioning could be handled consistently and within AMS, without altering anything in connection with the CR API or datamodel. That is to say that the attribute itself is a CR object but the value is not. As we discussed the other day there is no need for AMS managed attributes and values to be stored in the CR standard way with typed tables. In fact the typed tables design does not seem to lend itself to this idea of allowing users to specify data structures within applications.
There is another reason for needing the attribute to be an object which has become clearer whilst writing the Contacts module requirements document. Contacts will need to support personal contact lists, party contact lists and a global contacts list (which will include registered users). We want to avoid lots of duplication of information and take the opportunity to support collaborative data maintenance and updating by having features that verify data wherever a contact appears in more than one party's list. There will be certain fields that one person may keep on a contact that another person does not have or has not been given (a personal cellphone number for example). To support the sharing of data whilst maintaining privacy we need to have permissions set at the level of the attribute so that the system knows which attributes it can show to which users/parties.
Another point of clarification - I know there is already a tcl API for the CR and we are not suggesting re-inventing that. What is meant in the requirements document is that AMS should abstract the developer from the need to create CR objects when using custom attributes via AMS. AMS should take care of everything so that all you have to do is specify your attributes, give AMS your data values and ask for your data values back when you need them
Having said all of that, I still think that we must all agree exactly WHAT we want to build first before we get too involved in this kind of detail. I know it is slightly circular because to some extent it helps to consider the possible ways of doing things when discussing the requirements, however I firmly believe that we will benefit in the long run from writing the objectives down.
You mentioned that you would like to see a more concise and concrete document. I agree that that is important, and so did try to be as concrete as possible, to keep the requirements list short and to give specific examples, for two applications that are actually in progress, of what is needed from AMS (the use case examples). Could you perhaps give me some idea of which requirements you regard as important and which you do not think should be pursued.
Best Regards
R.
"- A notation and API for defining attributes and records where a record will be an OpenACS CR ‘record’ object that has child objects that are attributes (in this way AMS can support user defined complex compound data types analogous to the current OpenACS standard for ‘typed’ data in the CR)."
What is an "attribute" in this case?
I think we need some definitions of your use of the terms since we had quite a bit of confusion when we just said "attribute" in the chat just among three people.
Also note, Don and I at least, do want this work to go back into OpenACS so that is one reason we are more interested in the actual design of these features.
- Provide a Tcl API to define object_types
- Provide a user interface to define object_types
- Provide a Tcl API to retreive attributes of an object or list of objects
- including a consistent Tcl API and UI for handle regular acs_objects and content repository objects
- Provide a Tcl API to generate form specifications for data entry based on an object type definition
- Provide a user interface to define form properties for an object type definition
- optional Provide a Tcl API to generate a list specification based on an object type definition
In addition to these requirements, which manipulate object type definitions, AMS also needs to provide a way to define user-level attributes. This is mainly seen in the contacts application where each user may define custom fields to record for their contacts. This requirment may be different completely from the requirements to handle object types and standard objects.
Yes I agree, confusion reigns supreme!
"A notation and API for defining attributes and records where a record will be an OpenACS CR ‘record’ object that has child objects that are attributes (in this way AMS can support user defined complex compound data types analogous to the current OpenACS standard for ‘typed’ data in the CR)".
I meant attributes in the RDBMS sense NOT the OpenACS object model sense. It would probably be much clearer if I said the following:
An 'ams_record' will be a content type (a special one rather like a CR folder) which will have associated with it (as children) one or more of the CR content type 'ams_attribute'(s), which themselves will have a single attribute - a value.
I hope that removes some on the linguistic entanglements. :o|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.......some discussion about the object model.......
rickyh I think I now understand where we are at crossed purposes.....
rickyh Matthew and I were thinking primarily in terms of building something that allows us to store arbitrary data for existing cr_objects. I think that you have been driving towards an even more fundamental development which would allow for manipulation of the oacs object model. Am I correct?
daveb right.
daveb and what we thought we were disucssion was how to use the AMS ui to do that.
daveb :)
daveb maybe these things don't go together.
daveb you are almost defining the attributes on a per object basis.
rickyh Exactly! These last few lines need posting in the forum - I will do that. I have to disappear for 10 mins, is it ok with you if we continue after that for a few minutes. It is just that we should consider whether we should or should not do this all in AMS.
daveb right
It seems to me that attributes should be designed from the most practical uses, namely an attribute (or set thereof) as components of a user created table that is an OpenACS object. The table could have characteristics of a spreadsheet, where each attribute is a referenceable spreadsheet cell, and each cell has these (suggested) properties:
cell.value
cell.format
cell.style (a css reference)
cell.forumla
cell.order_of_operation
cell.reference (an abbreviation or maybe rc ref)
cell.x_location
cell.y_location
cell.name (optional)
cell.pretty_name (optional)
cell.description (optional)
where optional items are mainly useful for tables that consist of one cell size.
To make this work, there would need to be a table of spreadsheet objects, maybe with these properties:
table.name
table.pretty_name
table.description
table.format (css page reference)
table.columns (width)
table.rows (height)
table.orientation (default = horizontal)
table.key_prefix
The table.key_prefix would be a prefix added to user defined table names to prevent schema collision and isolate the api from accessing other tables.
This method would take advantage of existing user understanding of spreadsheets etc.
There are a couple of ideas overlapping here.
Defining attributes, either custom or object attributes including datatype, form widget, and display properties of an attribute.
Building a form of those attributes.
Displaying attributes.
So I think there is some reusable ideas from AMS and object_type definitions. Lee Denison was working on the object_type stuff and Matthew on the AMS attributes. Let's see what we can come up with after they can comment on our dicussion of thier work :)
I can certainly see value in having a module that provides 'spreadsheet like' facilities, however I don't think that it is appropriate here. In my opinion any attribute and/or object management system should behave like a database rather than a spreadsheet (i.e. I personally would oppose the use of cartesian co-ordinates for accessing information).
Nevertheless aspects of your nomenclature could be employed in the tcl API and if a spreadsheet like module was to be developed it would be much easier to build it on top of the services that we are trying to specify here.
R.
Yeah, I don't think that a spreadsheet package (SSP) should be part of the kernel, but I would be pleased if the AMS idea considers extreme cases like this. Some spreadsheets have permissions defined on a cell by cell basis (or by row).
The above spreadsheet model does not require cells to be referenced by x and y coordinates. Each cell could be referred to by it's name. Similarly, each row or column could be defined with a specific data type, essentially creating a "fuzzy" database. This is where the function of the spreadsheet package meets AMS. Both are capable of extending objects, just in a different way. This spreadsheet approach is going to be used to extend product data (beyond the custom fields features) in the ecommerce package. Each product type will share a spreadsheet of specifications for comparison purposes.
As I see it, the idea for AMS came about for associating scalar attributes. SSP is an expansion on that idea (as I see it). For an SSP, think JotSpot.com that can handle tables (maybe it does). I'll be glad to elaborate, but that's OT here. Anyway, just trying to think of AMS cases..
cheers,
Torben
As I understand the OpenACS object system, currently to add permissions control etc to the SSP for each cell of a spreadsheet, one would add cell.object_id and table.object_id to the above properties, where cell.object_id would inherit from table.object_id. Cell level control becomes important, for instance, with bookkeeping data.
Yet, creating each cell as an object would be overkill, because it's essentially an attribute of the table.object_id. Perhaps the AMS would be an appropriate use here, changing cell.object_id to cell.attribute_id or cell.attribute_type... whatever.
Maybe this is.. probably is.. way out there. I'll be quite now.. =)
Thanks Richard, Dave and folks for imposing some structure and starting to pin down the requirements for this stuff. Of the requirements discussed so far, here is the set, subject to agreement with Matt and the community, I would like to work towards, in the first release of the combined ams / dynamic-types package:
- Attribute management as a service - Principally we are supporting developers who want to use dynamic types and metadata driven forms in their own packages. Specifically we are supporting:
- Developers of 'empty vessel' applications - eg. a threaded discussion package which manages a hierarchy of messages but doesn't say whether each message should have a title, description, icon, attachment, etc or not. Instead these attributes of a message can be added dynamically after the application has been instantiated.
- Developers of fixed datamodel applications who want to store data in the content repository without knowing the details of content_items and content_revisions or would like the system to take care of add / edit pages using metadata driven forms.
- Extending (adding attributes to) existing types and creating new types - Whilst AMS doesn't support creating types from scratch right now, I got the impression from Matthew that he is keen to add it since all it really requires is to be able to store an object_id somewhere for each object instance. My dynamic-types package also supports creating new types. So, if I haven't mis-represented Matthew, we should aim for a combined package that supports both creation of new types and extensions of existing ones.
- Support both acs_object and content_revision - Allow developers to create new subtypes of, or extend existing subtypes of, both acs_object and content_revision through the same api. However, I would specifically like to exclude the following, because of the impact I forsee on the simplicity of implementation (ie. so that we maintain the versioning relationship of content_item and content_revision as it stands currently):
- Package developers may not version attributes independently of each other - when a new version of an attribute is created it must have a corresponding new version of the object.
- Versioned attributes cannot be added to non-versioned object types.
I would interpret these requirements to imply the following functionality:
- Provide a Tcl API to define object_types
- create / delete object type
- add / remove attribute from object type
- Provide a Tcl API to define form metadata for an object type
- create / delete form for object type
- add / remove widget from a form for a given attribute of an object type
- Provide a Tcl API to create and represent common combinations of form widget and datatype
- create / delete a template to represent a datatype / widget combination (eg. boolean / radio button / options {{Yes y} {No n}})
- Provide a Tcl API to add form elements to a form based on an object type definition
- Provide a Tcl API to process form submissions to create or update dynamic object type instances
- Provide a Tcl API to retreive attributes of an object or list of objects
Ideally I'd like to get those basic functions covered before looking at requirements like:
- Permissions on object type metadata (ie. controlling whether someone can add / remove or edit the attribute metadata of a type)
- Supporting composition relations (ie allowing an instance of a different object type (eg. address) to be an attribute of your new type)
- Allowing modification of attribute metadata, along with deprecation (ie. moving and attribute from one type to another, or changing the datatype of an attribute)
- Supporting object templates and list builder
- Supporting multi-valued attributes
Richard and I just had this IRC discussion to try to sort out where the overlap is between mine and Matthew's work. I think we established three principle differences in our requirements:
- dynamic-types supports both creating new types and extending existing types - AMS is focused on extending existing types
- dynamic-types does not support permissioning individual attribute values - AMS does
- dynamic-types does not support versioning individual attribute values - AMS does
How our implementations address these differences will dictate how much overlap there is in our code.
The major thrust of the differences is that Lee's work was predominantly based on the existing acs_object model and was to allow for the creation of 'typed' attributes (complete with associated tables [content_type] and [content_type_revisions] and associated views [content_typei] and content_typex].
AMS, in contrast, was conceived out of a 'skinny table' implementation that allowed for information (now understood to be called 'generic attributes') to be associated with any object_id. We were intending to do this using the existing data model to the greatest extent possible.
(References below are to previous post)
1. There had been talk of AMS allowing for the creation of new CR content_type(s) but this seems to overlap with Lee's work. We therefore propose to elminate this from the AMS requirements spec. Any 'generic attributes' (I think 'non-typed attributes' would be a clearer description) would be stored under the control of AMS under pre-defined AMS CR content types.
2. There seem to be good use cases for having the facility to support permissions at attribute value level. The contacts application is one such example (see contacts requirements doc for full details - should be finished tomorrow!). This would be cumbersome to support using 'typed attributes' and so it is proposed that this be one of the main distinguishing features of 'non-typed attributes' versus 'typed attributes'.
3. There also seem to be good use cases for having the facility to support revisions at attribute value level. Again, the contacts application is one such example. This would also be cumbersome to support using 'typed attributes' and so it is proposed that this be another distinguishing feature of 'non-typed attributes' versus 'typed attributes'.
This clear delineation of function should avoid confusion, duplication and overlap in the underlying functionality.
There is agreement that there is very likely to be overlap and opportunities for re-use in the user interfaces and the code that generates input specifications for ad_form and perhaps list builder. This provides the best opportunity for saving work and time through code re-use.
We have come to the conclusion (provisionally subject to broad agreement) that the existing permissions system does not lend itself to being stretched to support permissions by attribute value, because this would require every data value to be an object in the acs object model. There is discussion needed to establish the most elegant method of supporting this feature.
If these conclusions and proposals receive support, AMS will form an alternative 'light weight' method for developers to add user customisability to applications with developer defined content_types.
Lee's work will provide the ability for developers to create 'empty vessel' applications that are competely configurable. the advantage of having both systems is that in future, once an 'empty vessel' application has been defined it would be possible for the developer of that application to allow users to add customised 'non-typed' attributes to their content objects with permission control at the value level.
I think that we have now fully characterised the issues and largely resolved them. I hope very much that tomorrow we can finalise the AMS requirements document and progress with designing the data model.
Thank you very much everyone
R.
https://openacs.org/storage/download/index?version_id=215622
Feedback, suggestions, refinements, clarifications please...
http://www.openacs.org/storage/view/proposals/ams/AMS_Requirements.htm
The customized permissions of attribue values aspect of this doc wasn't discussed much, in part because we haven't decided exactly how this will happen. This will be fleshed out later when the functionality is built into contacts, and for those of you that are worried, it will not create objects for each attribute value - it will be simpler than that (programatically as well as in terms of efficiency). That ability is sometihng that is more of a long term goal for contacts so we can work with the current model first and figure it out when things are more stable.
I hope to implement much of this before I leave for Christmas... although I'm launching a new site between now and then so that might be dreaming :)
I just wrote this to Matthew in an email, but here it is again for the record:
You write in the specs that you don't want to sacrifice performance (max. 10%). However, I once tried to query a list of 1000 users with four "dynamic attributes" stored in a "skinny table", and the query took some 5 seconds on PostgreSQL. That's a lot, compared to the 0.2 seconds the query took before, and simply too much for a page that is frequently visited. And "serious" online communites may have 100.000 users and more...
So I firmly believe that you would need to provide a "storage type" to store attribute values together with the object's main SQL table. I think that acs_attributes already provides such a mechanism, but I'd have to check this...
Bests,
Frank
Thanks for the input. The issues that you raise are important and we have been careful to consider them.
We agree that there are always scaleability issues with the introduction of a join to a table that has lots of rows. There are however ways to implement this kind of attribute storage that avoids a join and the resulting performance hit.
One of the most performance hungry examples of using self-joined tables is storage of any hierarchy. A hierarchy trace 'on the fly' can be terribly slow. But even so, in a situation where queries are frequent but storage is not a problem and updates are infrequent, it becomes workable to use triggers to generate intermediate tables that store all descendents of each object for quick retrieval. I believe that the openacs permissions system uses something akin to this method and this works well.
In AMS there will be no hierarchy stored, and the selection of the attributes can be done as a 'grab me all attributes with this object_id' query. With appropriate indices, there is no reason to suspect that this will pose an unacceptable performance problem and initial tests (including the first version of contacts which used this scheme) work very quickly with several thousand rows.
Even the use of the 'ams_list' that groups attributes together in what could be described as 'virtual data rows' can be implemented by using an ams_list_id field in the main skinny attribute value table so that no join is necessary to the table that stores the details of the ams_list(s).
As a general principal, we will work towards 'munching' only integer keys only to gather the necessary ids which we will then go to the attribute values table with in a single (ideally) unjoined query. It is quicker to extract a result set of keys and then fetch their human readable values as a seperate step, than to join everything together and do it all in one query.
This makes sense to us but if you think we are mistaken or have missed the point, please do let us know.
Thanks for reviewing the document - your input is much appreciated.
Regards
Richard
thanks for the quick reply. We've been checking acs_attributes a bit more in detail, and the way ad_form lists and the extension tables are generated there. For us that looks very promising, because these plain extension tables are easy to add to an incredible number of existing "legacy" objects that we have in our systems (more then 100...), they are easily joined in SQL selects using a "select e.* from ... extension_table e" and they should have a good performance.
To put it the other way around: The "CR attributes plus trigger" option that you mentioned is definitely not going to work for us. We would need the "type_specific" storage.
However, the "type_specific" storage should fit nicely into the AMS, shouldn't it? It's really just a storage type, compared to the "cr" storage type that you proposed. And it is even already implemented in the OpenACS core.
So do you think that might be an option for you?
In the postitive case we would start working on the Oracle version nearly immediately, taking your current AMS code as a base and introducing what's necessary for the "type_specific" storage.
Bests,
Frank
We have been around this loop once already. Please refer to the AMS Requirements document again and you will notice a reference to OMS - the planned Object Management System or 'Dynamic Types' as it may be called.
This is what you will need - when it is done. We have been careful to eliminate overlap.
Regards
Richard
thanks for the reply. Yes, I've seen that you have mentioned the OMS in "the specs".
So it looks as if we would then have to go ahead with our stuff, because we need a rasonable "extensible architecture" within the next few weeks for a new project: https://openacs.org/forums/message-view?message_id=239510
According to our current roadmap, we're probably just going to add some extra functionality to the existing acs_attributes mechanism:
- Extend the "acs_attributes mechanism" to objects different from subtypes of "group"
- Add some suitable maintenance screens
- Add the AMS widgets to acs_attributes, probably using the ams_attributes table
- Add some absolute/relative positioning to ad_form
- Maybe create a simple "form designer" to store ad_form fields plus absolute/relative positioning in the database
Our main focus will be to create a production-ready system ASAP. This means that we will not necessarily priorize "generality".
I understand that these extensions are more or less orthogonal to the AMS functionality, right? So by using
the AMS table structure we should be pretty much prepared to integrate these two development branches in the future, is that right?
One importante point though: We will need to remove any dependency on the "postal-address" and the "ref-*" modules from the AMS, so we are going to create a branch of the AMS, based on your current version.
Bests,
Frank
Hi Folks,
Sorry I haven't been making much noise recently, I've been bogged down with internal project pressures.
I've made some progress with dynamic types and it is functional in postgres apart from some small remaining bugs. I haven't yet ported all the queries / plpgsql to oracle.
Here is a temporary link to a tarball of the code if you want to do anything with it.
Here is a temporary link to a basic CRM package which uses the dynamic types code. It has project specific code in it which I haven't had time to take out. I'm not sure whether it will install on stock OpenACS but the code which uses the dynamic types package is generic. You can find it in the tarball in crm-ui/www/admin/demographics/* and crm-base/lib/extended-info/*
