I want to share with you some toughts on the possibility to create a package dedicated to event triggering and handling by and from webpages.

The goal of this package would be to replicate most of the features of Unix inter-process comunication, with signals emitted from one page handled by one ore more other pages.

The use-case scenario in my idea would be this: new javascript libraries, togheter with recent web features and techniques allow the programmer to create "sockets" from javascript code running in the browser. Such sockets can be used to receive data pushed from the server, instead of being pulled by the browser.

Using events, one could open a socket to a webpage which would stay opened indefinitely. This page reacts to events triggered by changes into the server (like inserts, updates or deletes into database tables for example) and for each one pushes data to the connection, which could be used to update the gui or notify the user that an important event happened.

I have already done some succesful experiments leveraging the aolserver built-in event mechanism, togheter with jobs and queues, to create a webpage capable of reacting to a set of events, but I would gladly collect some opinion about the usefulness of a package like this before I go on. I haven't found something similar in features in the OpenAcs/Aolserver community, but maybe someone could point me in the right direction...

I want the server to notify connected clients about events happened. This means that wherever something noteful happens, the server should be able to push this information to the clients listening to such events.

I don't think it can be done with callbacks, because we cannot contact clients from the server directly. We need them to connect to a special webservice which remains silent until the event happens, then spits out some information.

In the meantime I created a package in which I implemented my idea of event handler: it allows the throwing of arbitrary events with a simple api. You can then define a webpage which will listen for any of this events to fire, each time reacting with a user-defined piece of code.

I will soon submit my code to the public to be commented.

This technique (set of techniques actually), also called long polling is one of the most exploited to achieve server push in the Web.

With HTML5 this is going to be replaced by websockets, and Claudio already pointed to an Aolserver implementation, but sadly, Internet Explorer hasn't adopted this feature already.

That said, Comet remains a low-tech, cross-browser alternative to server push, so I didn't waiste my time!

I improved my code so it won't leave neverending threads waiting in the server. A timeout parameter is now available, which will determine the lifetime of the handler.


Use it as you please!

I looked at your references. You spoke about a patch required to implement efficient comet features into Aolserver. Is that patch still required nowadays?

In your post about delivery threads you said that ad_returnfile_background exploits delivery threads. In Debian 7 version of OpenAcs (5.5) this proc exists and is set to send files in background if the ns_conn driver is nssock, which is true for my installation. Does it mean this patch is now part of the standard Aolserver? I would think so, as I can also find "ns_conn channel" into the API...

I have installed the Chat package, which uses your xotcl-core Chat class to implement a chat application. Big code in there Is that the right package to see your comet techniques in action?

I upgraded my test server to run the latest official version of OpenAcs (5.7) and I installed from the repository the latest available packages for xotcl-core (0.124) and xowiki (0.144)

Then I created a chat page as you explained and tried it out: the indicator in the browser keeps spinning and there seems to be no communication going on in the chat, even if I try to type something. I guess this is the problem you were speaking about, as the ajax requests to the "chat" page don't seem to be chunked if I inspect the headers...

Also I have not found any reference to chunked encoding in xotcl-core...

Do I have the latest available version? I've known from Claudio Pasolini that further development is happening which could not be reported to the mainstream yet.

As a temporary hack do you think it would be sufficient to set the encoding as chunked in the headers to return a chunked response and make chat work?

Do you think it would be ok to submit your patch to the Debian project so they can merge it with the standard? I could take care of it for you.

Now I am going to study and use your comet solution to implement real time notifications on my portal. I'd like some mean to specify custom comet webservices which would spit out info on an event basis, with the least possible relation with the client application (a generic json encoding). There seems to be all the tools to achieve this in your package, I only need to get one level down from the chat application and build something more generic.

If this would ever have the form of something reusable, I would gladly sumbit it to your opinion.


Thanks for everything Gustaf, enjoy your vacation!

your fine streaming comet implementation, which I clearly prefer over the others, was almost ok. It had a minor issue on Chrome because this browser refuses to expose partial response to ajax. After a lot of fiddling and searching I found out it could be solved by setting the response type as "octet-stream".

The problem then was that some special char was corrupted passing through the raw stream. I went over it by inserting some escaping/unescaping on the javascript side.

I can confirm now proper functioning of chat with this method on Linux versions of Firefox and Chrome, and also on Konqueror/Webkit, which is quite a choosy browser.

Note: the change of response type i put takes place in xotcl/bgdelivery-procs. Any side effect with that?

Will soon give more reports!

Please send me you modification, i'll update cvs....

many thanks and all the best
-gustaf neumann

Many thanks for testing and providing the changes, which are merged now into CVS head [1]. Please double-check, if everything is still fine. I have some mixed feelings about making the streaming versions default. The check for the browsers applies just for current versions (older versions behave differently), and streaming behavior might be influenced by a reverse proxy.

However, i have left streaming defaults as in your patch and added a longer comment session to the chat-procs with some warnings. This way, for streaming works immediately for people who want to try out COMET streaming without much hassle, and people working with large production sites know their requirements and will pass the appropriate "-mode " flag to Chat.

Many thanks and best regards
-gustaf neumann

[1] http://cvs.openacs.org/changelog/OpenACS?cs=MAIN%3Agustafn%3A20130217114806

i see the problem: when you put the chat into the master template then every user/client will become a member of the chat. The back-channel wont be closed, the server can't detect precisely, what connection should be reaped. Sending spaces (as in your changes) is not a good idea either, since one should keep the communication transparent. Furthermore, one has to cleanup not only the stale sockets, but as well the nsv-variables keeping the per-chat context. Doing the sweep over all chat clients for every message does not scale well.

What i have done now is to re-factor the subscriber logic a little to reduce potential code duplication and to call the subscriber sweeper from the (already preexisting) chat sweeper. The chat sweeper has configurable sweepinterval. The changes are in CVS head.

For watching the open files, i would recommend munin [1], for which i have written a set of plugins [2] mostly for naviserver, some of these work for aolserer as well. The set of plugins contains as well a monitor for open files to avoid surprises, when one is running out of file-handles (see sample view [3]).

I did not have the chance yet to look at last javascript/character coding changes.

best regards
-gustaf neumann


[1] http://munin-monitoring.org/
[2] git clone git://alice.wu.ac.at/munin-plugins
[3] http://openacs.org/xowiki/file/naviserver_production_lsof-month.png
[4] http://cvs.openacs.org/changelog/OpenACS?cs=MAIN%3Agustafn%3A20130221123133