Forum OpenACS Development: Toughts on an Event Engine

Posted by Antonio Pisano on 01/18/13 04:54 PM

Hello everyone,

I want to share with you some toughts on the possibility to create a package dedicated to event triggering and handling by and from webpages.

The goal of this package would be to replicate most of the features of Unix inter-process comunication, with signals emitted from one page handled by one ore more other pages.

The use-case scenario in my idea would be this: new javascript libraries, togheter with recent web features and techniques allow the programmer to create "sockets" from javascript code running in the browser. Such sockets can be used to receive data pushed from the server, instead of being pulled by the browser.

Using events, one could open a socket to a webpage which would stay opened indefinitely. This page reacts to events triggered by changes into the server (like inserts, updates or deletes into database tables for example) and for each one pushes data to the connection, which could be used to update the gui or notify the user that an important event happened.

I have already done some succesful experiments leveraging the aolserver built-in event mechanism, togheter with jobs and queues, to create a webpage capable of reacting to a set of events, but I would gladly collect some opinion about the usefulness of a package like this before I go on. I haven't found something similar in features in the OpenAcs/Aolserver community, but maybe someone could point me in the right direction...

2: Re: Toughts on an Event Engine (response to 1)

Posted by Claudio Pasolini on 01/20/13 06:33 PM

Hi Antonio,

the canonical way for inter package communication should be acs-service-contract. Personally I find it too complicated and think that using callbacks is way more simple.

Regarding the websockets Wolfgang Winkler made a Tcl implementation.

3: Re: Toughts on an Event Engine (response to 2)

Posted by Antonio Pisano on 01/21/13 12:06 PM

The use case I had in mind is slightly different from what it can already be achieved by callbacks. Let me explain:

I want the server to notify connected clients about events happened. This means that wherever something noteful happens, the server should be able to push this information to the clients listening to such events.

I don't think it can be done with callbacks, because we cannot contact clients from the server directly. We need them to connect to a special webservice which remains silent until the event happens, then spits out some information.

In the meantime I created a package in which I implemented my idea of event handler: it allows the throwing of arbitrary events with a simple api. You can then define a webpage which will listen for any of this events to fire, each time reacting with a user-defined piece of code.

I will soon submit my code to the public to be commented.

4: Re: Toughts on an Event Engine (response to 1)

Posted by Antonio Pisano on 01/21/13 09:13 PM

I created a GitHub repo and committed my package.

The url is this: https://github.com/Elettrotecnica/OpenAcs-Event-Handler

My event handler is entirely based on plain Aolserver API, with the only exception of the ad_proc syntax.

In the www folder of the package you can find a simple example of the package at work. One is the event creator, the other is the event handler. The first keeps firing events, while the second reacts to one or more of them executing a piece of code.

On the server side, this mechanism is totally agnostic regarding websockets or other advanced web techniques, just a lean and mean HTTP response.

Any comment will be much apreciated!

5: Re: Toughts on an Event Engine (response to 4)

Posted by Brian Fenton on 01/22/13 10:16 AM

This looks really cool! Well done Antonio. 😊

6: Re: Toughts on an Event Engine (response to 5)

Posted by Antonio Pisano on 01/22/13 12:00 PM

Thanks Brian!

I updated the code to allow the specification of a different script for each event caught. This way the mechanism should be more flexible.

The examples in www exploit the new feature.

7: Re: Toughts on an Event Engine (response to 6)

Posted by Antonio Pisano on 01/24/13 09:28 PM

Well, looks like I re-invented the wheel and my idea already has a name: Comet.

This technique (set of techniques actually), also called long polling is one of the most exploited to achieve server push in the Web.

With HTML5 this is going to be replaced by websockets, and Claudio already pointed to an Aolserver implementation, but sadly, Internet Explorer hasn't adopted this feature already.

That said, Comet remains a low-tech, cross-browser alternative to server push, so I didn't waiste my time! 😊

I improved my code so it won't leave neverending threads waiting in the server. A timeout parameter is now available, which will determine the lifetime of the handler.

Use it as you please!

8: Re: Toughts on an Event Engine (response to 7)

Posted by Gustaf Neumann on 01/25/13 10:01 AM

We have a comet implementation in OpenACS+Aolserver since a few years which allows to keep a push channel to potential multiple clients open. See:

https://openacs.org/forums/message-view?message_id=423438
http://www.openacs.org/xowiki/tag/COMET

all the best
-gustaf neumann

9: Re: Toughts on an Event Engine (response to 8)

Posted by Antonio Pisano on 01/25/13 06:09 PM

Thanks a lot Gustaf,

I looked at your references. You spoke about a patch required to implement efficient comet features into Aolserver. Is that patch still required nowadays?

In your post about delivery threads you said that ad_returnfile_background exploits delivery threads. In Debian 7 version of OpenAcs (5.5) this proc exists and is set to send files in background if the ns_conn driver is nssock, which is true for my installation. Does it mean this patch is now part of the standard Aolserver? I would think so, as I can also find "ns_conn channel" into the API...

I have installed the Chat package, which uses your xotcl-core Chat class to implement a chat application. Big code in there 😊 Is that the right package to see your comet techniques in action?

10: Re: Toughts on an Event Engine (response to 9)

Posted by Gustaf Neumann on 01/29/13 08:59 AM

The mentioned patches are included in aolserver and NaviServer since about 4 or 5 years. If you have e.g. aolserver 4.5.1, you are fine. The Chat packages is not needed for testing the chat functionality. It is easier, if you use just xowiki. Create a page of type ::xowiki::Object and use the following as content:

proc content {} {
  ::xowiki::Chat login -chat_id 22
}

When you render this page, you see the chat dialog. One can have multiple chats concurrently running. Every chat is identified by a chat_id. In the example above, the chat-id "22" is an arbitrary number. You might choose to use e.g. the package-id of xowiki as chat-id, but then you would permit a single chat per xowiki instance.

While testing, i saw that current browsers require chunked encoding for streaming HTML output. If the example above does not work for you in your installation, please upgrade xotcl-core to get the version with chunked encoding support.

all the best

-gustaf neumann

11: Re: Toughts on an Event Engine (response to 10)

Posted by Antonio Pisano on 01/29/13 05:15 PM

Thanks for the support Gustaf, but there is still something I don't understand.

I upgraded my test server to run the latest official version of OpenAcs (5.7) and I installed from the repository the latest available packages for xotcl-core (0.124) and xowiki (0.144)

Then I created a chat page as you explained and tried it out: the indicator in the browser keeps spinning and there seems to be no communication going on in the chat, even if I try to type something. I guess this is the problem you were speaking about, as the ajax requests to the "chat" page don't seem to be chunked if I inspect the headers...

Also I have not found any reference to chunked encoding in xotcl-core...

Do I have the latest available version? I've known from Claudio Pasolini that further development is happening which could not be reported to the mainstream yet.

As a temporary hack do you think it would be sufficient to set the encoding as chunked in the headers to return a chunked response and make chat work?

12: Re: Toughts on an Event Engine (response to 11)

Posted by Gustaf Neumann on 01/30/13 12:37 AM

The change introducing chunked-encoding is just in the bgdelivery procs. Have you got this patch?
http://cvs.openacs.org/changelog/OpenACS?cs=MAIN%3Agustafn%3A20130129073959

I have tested this with FF 19 and Google Chrome, but i don't have my full development environment, since i am currently on vacation in the colorado rocky mountains... But i am pretty sure, with the patch it will work.

all the best
-gustaf neumann

13: Re: Toughts on an Event Engine (response to 12)

Posted by Antonio Pisano on 01/30/13 05:40 PM

Yes, that made the trick!

Do you think it would be ok to submit your patch to the Debian project so they can merge it with the standard? I could take care of it for you.

Now I am going to study and use your comet solution to implement real time notifications on my portal. I'd like some mean to specify custom comet webservices which would spit out info on an event basis, with the least possible relation with the client application (a generic json encoding). There seems to be all the tools to achieve this in your package, I only need to get one level down from the chat application and build something more generic.

If this would ever have the form of something reusable, I would gladly sumbit it to your opinion.

Thanks for everything Gustaf, enjoy your vacation!

14: Re: Toughts on an Event Engine (response to 13)

Posted by Gustaf Neumann on 01/30/13 06:00 PM

i am not sure, how we manage the debian releases of OpenACS in general, but i think, the patch is quite local, so it should not cause unexpected side effects.

As you can see from the postings from 2006, i have tried various alternatives and the technique with the script tags worked out best. Several comet implementations seem to implement this now as well. The biggest limitation are probably that Tcl's event management via select() is practically limited to 1000 file handles, and it requires plain http communication (one would need a reverse proxy or similar to handle ssl/tls). These limits can be overcome, but these are not high on my priority list.

Generalizing is pretty straight forward, since the script tags simply fill up some json objects, which are handled by a javascript handler at the end.

let me know, when you have something.

all the best
-gustaf neumann

15: Re: Toughts on an Event Engine (response to 14)

Posted by Antonio Pisano on 02/06/13 12:35 AM

Dear Gustaf,

your fine streaming comet implementation, which I clearly prefer over the others, was almost ok. It had a minor issue on Chrome because this browser refuses to expose partial response to ajax. After a lot of fiddling and searching I found out it could be solved by setting the response type as "octet-stream".

The problem then was that some special char was corrupted passing through the raw stream. I went over it by inserting some escaping/unescaping on the javascript side.

I can confirm now proper functioning of chat with this method on Linux versions of Firefox and Chrome, and also on Konqueror/Webkit, which is quite a choosy browser.

Note: the change of response type i put takes place in xotcl/bgdelivery-procs. Any side effect with that?

Will soon give more reports!

16: Re: Toughts on an Event Engine (response to 15)

Posted by Antonio Pisano on 02/06/13 01:28 AM

Looks like there is not a way to make pure streaming work on Explorer. Opera too has issues, as it fires the onreadystateevent only once and would require continuous polling of the responseText...

Windows versions of Firefox and Chrome work fine. Safari on Windows is ok too, but as Konqueror, keeps showing the loading spinner even on pure streaming... well...

Anyway... looks like the best practice for comet streaming had already been achieved. Scripted streaming for Explorer, responseText polling for Opera (which I wouldn't mess with) and pure streaming for the rest.

I modified the regexp so it allows pure streaming for every browser except Opera and Explorer.

I can provide the modified files.

17: Re: Toughts on an Event Engine (response to 16)

Posted by Gustaf Neumann on 02/10/13 05:27 PM

Actually, "scripted streaming" should work on every common browser, including Opera.

Please send me you modification, i'll update cvs....

many thanks and all the best
-gustaf neumann

18: Re: Toughts on an Event Engine (response to 17)

Posted by Antonio Pisano on 02/11/13 01:08 PM

Yes, scripted streaming is the most portable technique, and it's not unelegant, but suffers from the "neverending spinner" problem. I like more the pure streaming solution because it is very "How it should be if world was a perfect place"...

As you already found out in 2006, seems like some sort of compromise is always required when you deal with server push on the web. In this years nothing really changed, and this will stay true until everybody (Microsoft...) will adopt newer techniques like websockets and similar, or implement their event handlers the way they should be.

Here is a dropbox link to the files.

http://dl.dropbox.com/u/7119685/gustaf.zip

Thanks to you for your support, I'm glad to give my little contribution.

19: Re: Toughts on an Event Engine (response to 18)

Posted by Gustaf Neumann on 02/17/13 01:01 PM

Dear Antonio,

Many thanks for testing and providing the changes, which are merged now into CVS head [1]. Please double-check, if everything is still fine. I have some mixed feelings about making the streaming versions default. The check for the browsers applies just for current versions (older versions behave differently), and streaming behavior might be influenced by a reverse proxy.

However, i have left streaming defaults as in your patch and added a longer comment session to the chat-procs with some warnings. This way, for streaming works immediately for people who want to try out COMET streaming without much hassle, and people working with large production sites know their requirements and will pass the appropriate "-mode " flag to Chat.

Many thanks and best regards
-gustaf neumann

[1] http://cvs.openacs.org/changelog/OpenACS?cs=MAIN%3Agustafn%3A20130217114806

20: Re: Toughts on an Event Engine (response to 19)

Posted by Antonio Pisano on 02/18/13 07:17 PM

Commit seems fine to me, the comment is very explicit and explains clearly how to choose the right method.

That said, I have some more modifications which I would like to expose and discuss with you:

in these days I managed to implement notifications on my portal using your Comet chat. It works like this: the master template included in every page calls the login on the chat, then waits for any message to occour (messages are sent by other pages).

The normal workflow of my users (the portal is an ERP) causes many and many calls of the login method, as there is little to no use of ajax and the master template is always called. Every time the chat logs in it opens a new channel.

After some hours this took to the reach of the 1000 file handles you were talking about and to a reboot of the server.

I dug some more into the code: when a message is sent to the subscribers, it is checked wether the channel is still alive, and if it's not it is closed. This works good in a proper chat, where messages are sent quite often. In my situation tough, this happens only seldom, as the notified events are not frequent, so dead channels are never cleaned.

I created a new method in the Subscriber class of bgdelivery, which takes care of cleaning dead channels every time someone subscribes. Similar methods exist for other objects in the bgdelivery, and seems like this solved the issue.

Also, I was not super happy about my previous change in the bgdeliveries-procs: after setting the content type to octet stream, the encoding of the characters had to be solved on the client side, causing extra effort to the end user. I found out I could just use the "encoding" command to translate the message server side, avoiding to use "escape" on the client.

I changed the chat page in xowiki too, only to allow the sending of messages containing html. If I don't allow it, it is impossible to send even the char "<", and it should be safe, because we already quote all the html in the message on the server.

I updated the zip file I linked to include the modifications I am talking about. The bgdelivery-procs file had been checked against your last cvs commit.

Let me know if it could be ok for you.

Best regards

Antonio

21: Re: Toughts on an Event Engine (response to 1)

Posted by Gustaf Neumann on 02/21/13 02:09 PM

Dear Antonio,

i see the problem: when you put the chat into the master template then every user/client will become a member of the chat. The back-channel wont be closed, the server can't detect precisely, what connection should be reaped. Sending spaces (as in your changes) is not a good idea either, since one should keep the communication transparent. Furthermore, one has to cleanup not only the stale sockets, but as well the nsv-variables keeping the per-chat context. Doing the sweep over all chat clients for every message does not scale well.

What i have done now is to re-factor the subscriber logic a little to reduce potential code duplication and to call the subscriber sweeper from the (already preexisting) chat sweeper. The chat sweeper has configurable sweepinterval. The changes are in CVS head.

For watching the open files, i would recommend munin [1], for which i have written a set of plugins [2] mostly for naviserver, some of these work for aolserer as well. The set of plugins contains as well a monitor for open files to avoid surprises, when one is running out of file-handles (see sample view [3]).

I did not have the chance yet to look at last javascript/character coding changes.

best regards
-gustaf neumann

[1] http://munin-monitoring.org/
[2] git clone git://alice.wu.ac.at/munin-plugins
[3] https://openacs.org/xowiki/file/naviserver_production_lsof-month.png
[4] http://cvs.openacs.org/changelog/OpenACS?cs=MAIN%3Agustafn%3A20130221123133

22: Re: Toughts on an Event Engine (response to 21)

Posted by Antonio Pisano on 02/26/13 12:09 PM

Thanks for having checked and solved the issue properly. Among the various changes, I see you now test the channel by reading instead than by writing... I fell a bit dumb for not having tought about it 😊

I am learning a lot about comet and xotcl thanks to you lately, but I have started thinking that maybe I am killing flies with granades here. The kind of notifications I need could also be achieved by the 'notification' package...

Anyway, we had a chance to refresh xotcl comet, so it had been worth something.

At your convenience, let me know about the remaining changes I sent you.

Best regards

Antonio