Forum OpenACS Development: Calling C library functions from an AolServer filter.

Hi,

I have the following doubt: which is the best way (performance and security) to call the functions in a C library from an AolServer Filter?.

Thanks a lot for the help!

Regards,

Jorge.

First of all, there is no difference between calling a C library function from a filter or elsewhere in the aolserver. Since aolserver 4.0, one can load arbitrary C-implemented tcl extensions, before that, it was required to implement aolserver modules. So, there is actually now no difference between calling a C function from Tcl or from the aolserver, except that you have to take care that your c-module is thread-safe (e.g. no global variables)

Most aolserver modules don't use swig, but i guess, one could use it in principle. i would however recommend to grab a simple c-extension and to add your functionality for that. Since tcl is string-based, it is quite simple to pass e.g. numbers, boolean, strings, lists etc, without the need for swig. See e.g. http://wiki.tcl.tk/2334 for a start.

Concerning recommended approaches: the speedup when comparing tcl and C based functions certainly varies depending on your applications. My experience is that one gets typically a factor of 7 by implementing in C. Using exec or open is not recommended due to speed and thread-safety (up to my knowledge there are still a few unsafe issues in exec). aolserver 4.5 has ns_proxy, which might be an alternative if you have to use exec.

hope this helps a little
-gustaf neumann

From the documentation of HDF
http://www.hdfgroup.uiuc.edu/papers/features/mthdf/ i see that the library achieves thread safety through serializing API calls. One can achieve a similar behavior via ns_proxy or libthread. This won't allow concurrent HDF calls, but should work otherwise.

One can load a C-implemented Tcl extension into the aolserver via the Tcl command load http://www.tcl.tk/man/tcl8.4/TclCmd/load.htm (or via package require, which uses load).

Uggg, you have a long way to go. I don't think there is much anyone can do to direct you. Let me at least try:

1. C libraries can be wrapped into an AOLserver module.
2. AOLserver modules expose one or more Tcl commands.
3. You can use these Tcl commands anywhere in AOLserver.

4. Filters are part of an HTTP request pipeline
5. Filters can use any (number of) available Tcl commands/procs.
6. Filters return a signal to the request pipeline:
filter_ok
filter_break
filter_return

7. There are three filter points:
preauth
postauth
trace

8. Registered procedures (at most one per request) fire after the postauth, and only if filter_return was not returned by a prior filter.

9. Trace filters always run.

10 None of this has any direct relationship to modules, modules usually just provide additional functionality.

Anyway, if HDF isn't wrapped in some kind of AOLserver module and exposed via a Tcl API, it is pretty useless. Maybe you could load it, but you couldn't access any of the functionality without the wrapper.

Jorge, what are you trying to use HDF5 for anyway? (It's conceivable that what you want to do can be accomplished some other way.) Also, just what is it you want to accomplish in this AOLserver filter, and why?

There is apparently tclhdf, which provides limited read-only access to HDF from Tcl. Depending on what that already has, it might or might not help you get started on wrapping the HDF5 C libraries.

AOLserver filters probably run in each connection thread. The simplest way to check it probably to put an ns_log statement into your filter, and then look in your server log to see what the thread identifier is.

I don't know what you actually want with your "buffer" question, you'll need to be more specific. A buffer or cache of what, for what? What are trying to do with it, and why do you want such a thing at all?

Given the scale of what you are trying to do, here is what I would suggest:

1. first use a 'trace' filter. These run after the connection has closed, they always run, and theoretically you have the most information about what happened, essentially you can record information during the request and handle it at the end. Trace filters don't interfere with the quickness of serving the request.

2. Dump you collected information to a log file of some type. You may be able to just piggyback on the error log, or the access log, or create a new file to accept output.

3. Unfortunately, 2 requires that there is some text format for the HDF. I bet there is one.

4. There are certain utilities that can filter the error log to multiple files (multilog), so using the error.log facility is a good choice.

5. Occasionally process the output files with whatever tools are available for your HDF client/server.

Usint a text file buffer, you can start immediately instead of creating a module specific to AOLserver or Tcl, and you can test each part separately. You also have a transaction log, and you don't have to worry if your HDF software errors out, or if you change software.

The text file is your buffer, with something like multilog, you can have more than one, so you only process the ones that are closed. This provides a separation of writing/reading from the file.

Please note that I have no idea what data you are collecting, but only that eventually everything starts as a string, and Tcl is very good at producing strings. AOLserver is even better at handling strings (including binary data).

But I did look at the HDF, which I found out stands for Hierarchical Data Format, there appears to be a significant amount of up-front data-modeling. If you continue to work on this, it would be very interesting to hear from you on how you use HDF.

The details of applying this to AOLserver (how to use this in a filter) are simple by comparison, so continue to ask for help. A good starting point is simply writing data to a file, even from a Tcl shell, that does what you want, and can be used by whatever downstream tools you choose, after that, adding that to AOLserver is easy.