Forum OpenACS Development: CR Advice

Collapse
Posted by Randy Ferrer on

Hello everyone - I have a financial information gathering/presentation system that I wish to integrate into OpenACS and I am wondering whether or not it is a good idea to build this on top of the CR or if it might be best to build this outside of the CR. It would seem that using the CR is desirable, but as I look at some projects built with OACS some have chosen to implement outside of CR (Unidos project for example). So being new to OACS, at this point I am a bit confused. 😟 I would really appreciate any guidance/opinions that any of you can provide. This is what I wish to implement:

Project Description

The project consists of gathering and presenting to subscribers data for various classes of financial instruments. These go from stocks and bonds to currency and options as well as other more exotic asset types. The idea is to implement a large range of asset classes over several projects, so the data model that I have built is extensible to support the addition of various financial instruments over time.

The data currently being gathered is quite extensive. For example, for fixed income instruments we are gathering, bid, ask, various yield types, various analytics including duration, mod duration and many others. Total number of data types stands at 50 over various bond classes. For stocks, there are about 25 data types.

The price/analytics data is dynamic ("end of day" as well as "intraday") while the information describing the assets is quite static. The number of records for intraday data will reach several thousand per day across all the various asset classes being contemplated since much of this data will be streaming into the system and updating every couple of minutes in some cases. The intent is to purge this data after summarizing it every couple of days.

The various instruments each have a unique identifier and each belongs to at least one group. The main groupings are regional. One grouping for example is Latam Bonds which groups together bonds belonging to countries in South & Central America.

Integration with other content types is a must. For example a macro data db will be implemented. It is critical that searches can be built using this data. For example - give me all corporate bonds for countries whose GDP > X and has < Y population. A typical publishing paradigm will also exist on the site where relevant stories will be available.

Viewers will be able to subscribe to one or more asset groups (data view - Latam for example) to view data for the group(s) they have chosen. Would it be best to handle permissioning for each "data view" via the permissioning system or would it be best to permission a user to have access to the "Fin Data Package" and then simply provide the view subscribed to without necessarily going through the permissioning/priviledges system for this?

Longer term, various applications will be built on top of this. For example a portfolio application.

My main concern right now, as I mention at the beginning, is if it is recommended that something like this be constructed using the CR or whether it is desirable to bypass the CR completely. It is critical to have full integration into the various OACS services like templating, search etc. One more goal - once finished, I intend to release the application under the GPL and make it freely available either as part of the OpenACS main distribution or as an add-on outside of the main distribution packages as the community thinks best. With this in mind, it seems wise to have the package as fully integrated into the OpenACS core as possible as I think this will make it more useful to the general OpenACS community. (right?) I look forward to any guidance anyone can provide on these issues. Many thanks!

Collapse
2: Re: CR Advice (response to 1)
Posted by Talli Somekh on
Leaving aside the tech question for a moment...

"One more goal - once finished, I intend to release the application under the GPL and make it freely available either as part of the OpenACS main distribution or as an add-on outside of the main distribution packages as the community thinks best. "

Randy, you rock!

talli

Collapse
3: Re: CR Advice (response to 1)
Posted by Jun Yamog on
Hi Randy,

The benefits of CR are

- support for versioning
- basic content types are there so you just need to extend them.
- a keyword/category service
- ability to relate multiple templates on a per item level
- ability to relate item A with several items in the CR
- organizing of items by folders
- and other benefits of OACS basic object model (permissions, etc.)

As I see it, it would be a good candidate for a CR app.  One thing that you should look into is the scalability of CR.  I have worked with CR but not to the extent of numbers of contents that you have described.  In case you have got a top team that is better than aD's team then either modify CR or build one from scratch.

Here is also a good read on the different versioning strategies that was used by aD, ACS 3.x, ACS 4.x and CCM 5.x

https://listman.redhat.com/pipermail/redhat-ccm-list/2003-February/000791.html

At any rate its great news to have another CMS in the works.

Collapse
4: Re: CR Advice (response to 1)
Posted by Jeff Davis on
I would put the instruments into the CR but not the time series data. The use acs_rels to map instruments to asset groups (which would also be in CR), permissioning should be at the asset group level (for read checks anyway).

for fast searches you would probably want a denormalized flat table with lots of indexes (maybe split into two tables, a largely static one and a mostly dynamic one which has things like change on the day and other realtime stuff).

you will also probably want to be careful how you lay out the timeseries database since you will want to balance the IO accross several devices if you are taking tick data. In oracle you would partition the table and split the indexes accross multiple devices but I don't know how you would handle it in postgres.

Collapse
5: Re: CR Advice (response to 1)
Posted by Don Baccus on
Best you can do in Postgres is to put index files on a separate device from the data file.  There's no table partioning available.  The management tool for putting index (or other) files on a separate platter is called "ln -s"

There will be a *sane* way of doing this in the future, maybe PG 7.4 ...

Collapse
6: Re: CR Advice (response to 1)
Posted by Randy Ferrer on

Thanks for your suggestions so far. The site will be developed on PG for now and yes - performance/scalability has been a bit of a concern for me, but I'll find out soon enough. I am going to take your suggestions Jeff and start as you suggest. I have much to consider as the requirements for the site are growing fast as opportunities emerge - who knows - maybe there's a "dotFin" in the works here. 😉

Randy

Collapse
7: OpenACS financial data app (response to 1)
Posted by Andrew Piskorski on
Randy, could you tell us something about what your goals and
motivations are for building this, how you hope to use it once it's
done, the business case driving the work, etc.?
Collapse
Posted by Randy Ferrer on
Hi Andrew - The current goal is to enable OACS to host communities interested in discussing financial matters such as best trading strategies, what a price trend means to a specific financial instrument etc. Said another way, one big motivator behind such a community is to promote flow of information which will assist community members to make better informed investing decisions. In my view,  fundamental to this process is enabling OACS to serve financial data (prices, analytics)to the community along with the more typical news and other financial reports. A second goal once data is enabled is to provide applications to the community allowing them to create portfolios, perform calculations etc. on the data provided. In my experience, I find that the financial "advice" most people trust is that of other people that are investors like themselves. OACS IMHO, has no parallel anywhere in terms of providing tools to enable communities to form and thrive. All that's missing is the tools to enable gathering/dissemination of financial data for any type of financial instrument. I hope this clarifies a bit. 😊

Randy