Forum OpenACS Development: Re: Toughts on an Event Engine

Posted by Antonio Pisano on
Yes, that made the trick!

Do you think it would be ok to submit your patch to the Debian project so they can merge it with the standard? I could take care of it for you.

Now I am going to study and use your comet solution to implement real time notifications on my portal. I'd like some mean to specify custom comet webservices which would spit out info on an event basis, with the least possible relation with the client application (a generic json encoding). There seems to be all the tools to achieve this in your package, I only need to get one level down from the chat application and build something more generic.

If this would ever have the form of something reusable, I would gladly sumbit it to your opinion.

Thanks for everything Gustaf, enjoy your vacation!

Posted by Gustaf Neumann on
i am not sure, how we manage the debian releases of OpenACS in general, but i think, the patch is quite local, so it should not cause unexpected side effects.

As you can see from the postings from 2006, i have tried various alternatives and the technique with the script tags worked out best. Several comet implementations seem to implement this now as well. The biggest limitation are probably that Tcl's event management via select() is practically limited to 1000 file handles, and it requires plain http communication (one would need a reverse proxy or similar to handle ssl/tls). These limits can be overcome, but these are not high on my priority list.

Generalizing is pretty straight forward, since the script tags simply fill up some json objects, which are handled by a javascript handler at the end.

let me know, when you have something.

all the best
-gustaf neumann

Posted by Antonio Pisano on
Dear Gustaf,

your fine streaming comet implementation, which I clearly prefer over the others, was almost ok. It had a minor issue on Chrome because this browser refuses to expose partial response to ajax. After a lot of fiddling and searching I found out it could be solved by setting the response type as "octet-stream".

The problem then was that some special char was corrupted passing through the raw stream. I went over it by inserting some escaping/unescaping on the javascript side.

I can confirm now proper functioning of chat with this method on Linux versions of Firefox and Chrome, and also on Konqueror/Webkit, which is quite a choosy browser.

Note: the change of response type i put takes place in xotcl/bgdelivery-procs. Any side effect with that?

Will soon give more reports!

Posted by Antonio Pisano on
Looks like there is not a way to make pure streaming work on Explorer. Opera too has issues, as it fires the onreadystateevent only once and would require continuous polling of the responseText...

Windows versions of Firefox and Chrome work fine. Safari on Windows is ok too, but as Konqueror, keeps showing the loading spinner even on pure streaming... well...

Anyway... looks like the best practice for comet streaming had already been achieved. Scripted streaming for Explorer, responseText polling for Opera (which I wouldn't mess with) and pure streaming for the rest.

I modified the regexp so it allows pure streaming for every browser except Opera and Explorer.

I can provide the modified files.

Posted by Gustaf Neumann on
Actually, "scripted streaming" should work on every common browser, including Opera.

Please send me you modification, i'll update cvs....

many thanks and all the best
-gustaf neumann

Posted by Antonio Pisano on
Yes, scripted streaming is the most portable technique, and it's not unelegant, but suffers from the "neverending spinner" problem. I like more the pure streaming solution because it is very "How it should be if world was a perfect place"...

As you already found out in 2006, seems like some sort of compromise is always required when you deal with server push on the web. In this years nothing really changed, and this will stay true until everybody (Microsoft...) will adopt newer techniques like websockets and similar, or implement their event handlers the way they should be.

Here is a dropbox link to the files.

Thanks to you for your support, I'm glad to give my little contribution.

Posted by Gustaf Neumann on
Dear Antonio,

Many thanks for testing and providing the changes, which are merged now into CVS head [1]. Please double-check, if everything is still fine. I have some mixed feelings about making the streaming versions default. The check for the browsers applies just for current versions (older versions behave differently), and streaming behavior might be influenced by a reverse proxy.

However, i have left streaming defaults as in your patch and added a longer comment session to the chat-procs with some warnings. This way, for streaming works immediately for people who want to try out COMET streaming without much hassle, and people working with large production sites know their requirements and will pass the appropriate "-mode " flag to Chat.

Many thanks and best regards
-gustaf neumann


Posted by Antonio Pisano on
Commit seems fine to me, the comment is very explicit and explains clearly how to choose the right method.

That said, I have some more modifications which I would like to expose and discuss with you:

in these days I managed to implement notifications on my portal using your Comet chat. It works like this: the master template included in every page calls the login on the chat, then waits for any message to occour (messages are sent by other pages).

The normal workflow of my users (the portal is an ERP) causes many and many calls of the login method, as there is little to no use of ajax and the master template is always called. Every time the chat logs in it opens a new channel.

After some hours this took to the reach of the 1000 file handles you were talking about and to a reboot of the server.

I dug some more into the code: when a message is sent to the subscribers, it is checked wether the channel is still alive, and if it's not it is closed. This works good in a proper chat, where messages are sent quite often. In my situation tough, this happens only seldom, as the notified events are not frequent, so dead channels are never cleaned.

I created a new method in the Subscriber class of bgdelivery, which takes care of cleaning dead channels every time someone subscribes. Similar methods exist for other objects in the bgdelivery, and seems like this solved the issue.

Also, I was not super happy about my previous change in the bgdeliveries-procs: after setting the content type to octet stream, the encoding of the characters had to be solved on the client side, causing extra effort to the end user. I found out I could just use the "encoding" command to translate the message server side, avoiding to use "escape" on the client.

I changed the chat page in xowiki too, only to allow the sending of messages containing html. If I don't allow it, it is impossible to send even the char "<", and it should be safe, because we already quote all the html in the message on the server.

I updated the zip file I linked to include the modifications I am talking about. The bgdelivery-procs file had been checked against your last cvs commit.

Let me know if it could be ok for you.

Best regards