Forum OpenACS Development: Re: Calling C library functions from an AolServer filter.

Collapse
11: Re: Calling C library functions from an AolServer filter. (response to 10)
Posted by Tom Jackson on 02/27/08 12:22 AM
Given the scale of what you are trying to do, here is what I would suggest:

1. first use a 'trace' filter. These run after the connection has closed, they always run, and theoretically you have the most information about what happened, essentially you can record information during the request and handle it at the end. Trace filters don't interfere with the quickness of serving the request.

2. Dump you collected information to a log file of some type. You may be able to just piggyback on the error log, or the access log, or create a new file to accept output.

3. Unfortunately, 2 requires that there is some text format for the HDF. I bet there is one.

4. There are certain utilities that can filter the error log to multiple files (multilog), so using the error.log facility is a good choice.

5. Occasionally process the output files with whatever tools are available for your HDF client/server.

Usint a text file buffer, you can start immediately instead of creating a module specific to AOLserver or Tcl, and you can test each part separately. You also have a transaction log, and you don't have to worry if your HDF software errors out, or if you change software.

Collapse
12: Re: Calling C library functions from an AolServer filter. (response to 11)
Posted by Jorge Couchet on 02/27/08 12:52 AM
Hi Tom,

First, thanks again for your help!!.

Your solution is close to my second alternative:

*****************
Another solution is to generate a kind of "ring buffer" (as the Data Turbine Initiative manages), the filter is sending the data to this "ring buffer", and the data from the buffer is saved to a disk. Another process is in charge to transform this data to the HDF format and load in the database the meta-data necessary to track it.
*****************

My only doubt is about the efficiency if is the filter that directly dump the information to the text file, because it'll be requesting I/O access per each request, and with a kind of "ring buffer", the I/O operations could be delayed until to have a pre-configured information amount (as for example 100 connections).

I like the idea to use a trace filter!

Regards,

Jorge.

Collapse
13: Re: Calling C library functions from an AolServer filter. (response to 12)
Posted by Tom Jackson on 02/27/08 02:56 AM
The text file is your buffer, with something like multilog, you can have more than one, so you only process the ones that are closed. This provides a separation of writing/reading from the file.

Please note that I have no idea what data you are collecting, but only that eventually everything starts as a string, and Tcl is very good at producing strings. AOLserver is even better at handling strings (including binary data).

But I did look at the HDF, which I found out stands for Hierarchical Data Format, there appears to be a significant amount of up-front data-modeling. If you continue to work on this, it would be very interesting to hear from you on how you use HDF.

The details of applying this to AOLserver (how to use this in a filter) are simple by comparison, so continue to ask for help. A good starting point is simply writing data to a file, even from a Tcl shell, that does what you want, and can be used by whatever downstream tools you choose, after that, adding that to AOLserver is easy.