Forum OpenACS Q&A: ]project-open[: Current thinking about AJAX, Sencha and AOLserver

1: ]project-open[: Current thinking about AJAX, Sencha and AOLserver

Posted by Frank Bergmann on 05/30/11 05:34 PM

Hi,

I'm writing this post because we (]po[) are at the point of adopting a completely new architecture for ]project-open[. It would be great to hear your opinion about that.

The following post already summarizes a lot of this thinking:
https://sourceforge.net/projects/project-open/forums/forum/295937/topic/4529158

The "happy new future world" would basically look like this:

- Apache + PHP
- PostgreSQL and the OpenACS data model
- A relatively small layer of REST data sources
- Loads of easily modifiable Sencha AJAX packages
- Easily installable on all important OSs
- All of this compatible with HTML 5 and mobile devices (Sencha Touch).
- Millions of developers familiar with the architecture

Sencha:

Thanks to Malte for proposing Sencha as a possible AJAX GUI for ]po[.
Basically:

- The Sencha libraries are "similar in spririt" to FormBuilder and ListBuilder. You specify the fields you want to show and a "data source". It's a declarative approach similar to the one taken in OpenACS. But a lot easier to learn...

- Sencha GUI development speed seems to be faster then FormBuilder and ListBuilder, at least after the learning time.

- JavaScript/Sencha development skills are a lot more common then TCL development skills. I believe that a lot more people (]po[ partners and customers...) will start modifying and adding code then before.

- Getting data out of OpenACS/]po[ is easy. We are using our generic ]po[ REST interface (intranet-rest) as a back-end. intranet-rest produces JSON data in a format directly compatible with default Sencha "store" models. Custom data sources can be implemented either via a piece of TCL, or using a custom "report" (=SQL query) that returns data in JSON format.

- There is a smooth migration path with little overhead: We start with rewriting the most important pages in Sencha while maintaining the HTML GUI for the more complex cases. Later we could get rid of the HTML GUI completely.

The implications of the new architecture could be huge:

- Reduce the developer bottleneck of OpenACS.
- Get rid of AOLserver. We would have to rewrite only the relatively small REST interface in PHP or whatever language.

Roadmap:

- We are currently working on a customer project using Sencha + intranet-rest in order to develop a user-friendly ticket tracker. In this project we are extending the generic REST etc.

- We've got a second project in the pipeline to develop an AJAX GUI for the generic ]po[ project management functionality.

- At some moment (no timeframe yet) we'll develop a PHP implementation of intranet-rest, so that the Sencha GUI will be able to work against Apache.

Difficulties:

Not much as far as I can see right now.
- We are currently using the FormBuilder "new.tcl" pages for saving and creating new objects from the REST GUI. We'll try to reduce the use of these pages to a minimum and move the functionality into intranet-rest.
- Sencha performance seems to be poor on IE6 and Firefox 3.x

It would be great to hear about your feedback.
- Did I miss something important?
- Is this migration path also an option for OpenACS?

Cheers!
Frank

2: Re: ]project-open[: Current thinking about AJAX, Sencha and AOLserver (response to 1)

Posted by Eduardo Santos on 05/30/11 08:27 PM

Hi Frank,

So, you've chosen to move basically most importants TCL codes to JavaScript? Interesting because I would not go that way in a first thought. I like JQuery better, but it's only an opinion.

I'm just curious about how you are going to replace server operations, such as the ns_* functions. I guess your approach is to design completely new interfaces, so you would not need to use AOLServer functions, right?

You would also have to change data manipulation functions, but I guess you're dealing this on REST package you've created.

I can't say if it's an option, but it would be nice if you could share the code you are developing now.

Congratulations for the initiative.

3: Re: ]project-open[: Current thinking about AJAX, Sencha and AOLserver (response to 1)

Posted by Dave Bauer on 05/31/11 02:17 AM

Hi Frank,

This is very interesting.

I have used ExtJs (the predecessor to Sencha) and it was definitely very challanging to develop an debug but very interesting. With the new mobile features it is very compelling.

Do you have any information on accessible use of Sencha? I found with the way ExtJs took over the entire DOM and created the entire UI in javascript to be slow, and hard to manage. It definitely could have changed since I last looked. It seems that progressive enhacncement is easy to develop accessible web applications, and also makes they very usable on mobile devices without any extra special code.

I appreciate any insight you can give.

4: Re: ]project-open[: Current thinking about AJAX, Sencha and AOLserver (response to 1)

Posted by Taras Mitran on 06/09/11 02:20 PM

We do a lot of Sencha+OpenACS work, so here are a few notes/suggestions:

-- Sencha Touch has some issues with many HTC android devices and many pre 2.2 android versions with things like scrolling performance. It also doesn't work (although I haven't tested - they don't claim support and I'm pretty sure it would fail to work) on Windows 7 phones (although the market share is tiny, Gartner thinks they'll pick up steam and Nokia will probably give them a big boost) nor several custom browsers (opera/firefox mobile) vs the native webkits on iphone/android. If you are fine with 85% of the mobile market in compatibility, then great - otherwise you would want to use jquery mobile.

Remember, both can take advantage of json data from a RESTful API, and you have to have a different front-end anyways for mobile, so its possible to use jquery mobile but Sencha's ExtJS.

-- I think it will probably be difficult to use openACS but not aolserver or at least not worth the trouble. It seems like switching from both at once would be less painful. What does aolserver do that you don't like?

-- If you want a really tiny REST interface that returns JSON, PHP is probably the most obvious choice, but you could also look at node.js (its javascript on the back end) as it makes that very convenient.

-- start your forms in Extjs 4.0 (vs 3.X) since they have trashed the form layout (now you just use regular layouts like vbox to position elements), as well as graphing being migrated to HTML5 from flash.

Your note here:

"- There is a smooth migration path with little overhead: We start with rewriting the most important pages in Sencha while maintaining the HTML GUI for the more complex cases. Later we could get rid of the HTML GUI completely."

Is exactly what we did. Its too big to replace openACS/aolserver, re-write all the code, etc. at once.

5: Re: ]project-open[: Current thinking about AJAX, Sencha and AOLserver (response to 3)

Posted by Taras Mitran on 06/09/11 02:26 PM

ExtJS can be pretty fast, but you need to manage your code better than progressive enhancement for sure. For example, minify+gzip all the code, long expiring headers, etc.

If you have an app that doesn't need to look like a desktop app (like windows xp's file browser or MS Office apps), you need to support users without javascript running (still like 2% in the US!), etc. then jquery is a better way to go.

They are not incompatible though - I use jquery for many things - even on the same page as extjs code (jquery is absolutely tiny and doesn't add any overhead).

6: Re: ]project-open[: Current thinking about AJAX, Sencha and AOLserver (response to 4)

Posted by Dave Bauer on 06/09/11 03:35 PM

Taras,

Can you discuss accessibility in relation to Sencha ExtJS? It seems that it is much more work to make that type of UI that totally builds the DOM from Javascript accessible.

If you have an experience with that type of thing it would be very helpful.

Thanks!

7: Re: ]project-open[: Current thinking about AJAX, Sencha and AOLserver (response to 6)

Posted by Taras Mitran on 06/14/11 02:05 AM

By 'accessibility', do you mean ARIA/Section 508 compliance like keyboard shortcuts, color themes visibility-impaired users, etc.?

Or you do mean accessible like being able to programatically access variables, objects, UI widgets and manipulate them like you would with jquery selectors/manipulation where the DOM structure is more traditional?

8: Re: ]project-open[: Current thinking about AJAX, Sencha and AOLserver (response to 7)

Posted by Dave Bauer on 06/14/11 02:43 AM

This one: " ARIA/Section 508 compliance like keyboard shortcuts, color themes visibility-impaired users, etc.?"

9: Re: ]project-open[: Current thinking about AJAX, Sencha and AOLserver (response to 8)

Posted by Taras Mitran on 06/18/11 05:57 PM

Our apps have been largely for non-government or general public usage and haven't had such compliance concerns, so I wouldn't be an expert on such matters, but they do have some examples:

keyboard navigation:

http://dev.sencha.com/deploy/ext-4.0.2a/examples/key-feed-viewer/feed-viewer.html

tree keyboard navigation:

http://dev.sencha.com/deploy/ext-4.0.2a/examples/tree/treegrid.html

You can create event handlers on pretty much everything - I don't create a lot of arrow-key handlers but I've written plenty of right click, enter-key, hover-over, etc. events and you can bubble those up for some more generic handling.

dark theme:

http://dev.sencha.com/deploy/ext-4.0.2a/examples/form/form-grid-access.html

but basically all the styles/colors/buttons are now css driven and don't rely on images (unless you use icons) so its pretty easy to theme.

10: Re: ]project-open[: Current thinking about AJAX, Sencha and AOLserver (response to 9)

Posted by Antonio Pisano on 01/07/13 04:51 PM

Hello everyone, I bump this old thread to share my opinion as it is quite strong and this is an important matter in nowadays Web.

I do think as you that client-side Ajax and REST webservices on the server side is the way to go for the web (and most of general purpose applications too, maybe), but I would definitely not strive to get rid of Tcl instead of PHP. Rather the countrary.

I find Tcl much more suitable to web development, in particular for its string manipulation capabilities and funcional-style features, such as executing a string as a piece of code in a very simple way.

Plus, OpenAcs is a perfect structure for developing webservices: cleanest interaction with db ever, auto-sanitizing of strings in queries, clean definition and checking of allowed page query variables, a clean and simple filesystem API, the adp templating system... And all this stays true even throwing away all of the HTML user interface features.

Programs in Tcl/OpenAcs are shorter and clearer, and all the Tcl you will ever use is in 12 short webpages... the rest is in the specific api-doc page which is auto-generated each procedure you declare.

The plus of PHP is... the user base.

Speaking about ExtJs: it is a very powerful tool, but I personally don't like it. I understand this is all a matter of taste, but I don't like frameworks which are too structured, because they always end getting in my way. While it certainly tries to hide and encapsulate everything, I found myself (and most of the people on the forums, even Sencha's) wrestling with the framework, with very unelegant results. Also, as we can see with ajax-filestorage package here on OpenAcs which is no longer functioning on modern browsers, you cannot rely on it on the long term, and this because it is a very complex and unstable library (in the sense of features, not plain bugs).

JQuery and it's extensions, make a more light and elegant, yet powerful choice. It is agnostic respect to your development style, saves you lines of code without really telling you how you should get things done, and can be engineered at will if you so desire.

A lot of strong opinions here. I want to make clear that all of this products are free software which I used and used to love, so I didn't mean to be disrespectful. This is just my technical opinion. If somebody would like to bring his/her point of view I would gladly discuss about it.

11: Re: ]project-open[: Current status about AJAX, Sencha and AOLserver (response to 1)

Posted by Frank Bergmann on 01/29/13 12:39 PM

Hi!

Sorry, I had notifications turned off, otherwise I would have answered earlier. Here is a summary of the current state of Sencha development in ]po[:

- Accessibility:
@Dave, I've got no information on this, sorry. Accessibility is not a priority for us.

- Reference Application:
We've developed a rather complex ticket tracking application for a Spanish gov site purely in Sencha. The result has been quite impressive. The most difficult issue was network performance, because the app requires to load the complete lists of customers and users into the browser. We can now work around this issue with a different architecture and nsreturnz, amongst other measures. We can setup a demo server if that's interesting to somebody.

- Embedding Sencha in HTML Pages:
We have created a number of Sencha charts as ]po[ portlets to be embedded into normal HTML pages. This works fine in general, with a certain delay between the load of the HTML page and the start of the Sencha app. The charts are very pretty, quite a difference to the rest of the ]po[ GUI... We've been playing around with using Sencha as a file-storage front-end now, and that also looks pretty good. However, we have no clear idea yet how multiple portlets on a single page will behave in terms of performance.

- License Issues:
Sencha Ext-JS is licensed under GPL V3 (or proprietary if you pay). However, GPL V3 code is really difficult to separate from GPL V2 and proprietary code if they are used in a single application. This is the reason why we have excluded any Sencha code from the current ]po[ V4.0 release. Instead, we are going to include an "app store" in V4.1, which will allow customers to dynamically load GPL V3 packages, thus allowing us to avoid any "license contamination".

- REST Interface:
@Taras: Our ticket tracker application above requires some 15 different object types via REST interface. So we have decided to use a generic REST interface using the OpenACS SQL metadata (http://www.project-open.org/en/package_intranet_rest). New OpenACS object types will automatically become accessible via REST, and you don't need to hand-code the interfaces. This greatly reduces errors and security/permission issues, because the code is reduced.

- ]po[ V4.0 vs. ]po[ V4.1:
We are releasing V4.0 at the moment, and the "release process" will still take several months (after the product is now getting ready for wide-spread use we need to work on PR and marketing...). V4.1 will probably take until summer 2014 or even later. Please see our roadmap for details: http://www.project-open.org/en/project_open_roadmap

- TCL vs PHP:
At the moment all back-end code is TCL. Maybe I've expressed myself a bit ambiguously. We don't _want_ to port all TCL stuff to PHP if we can avoid it. We just _need_ PHP as a fall-back option if OpenACS activity should go down even further. So it's a good idea to try to use the new ]po[ REST interface as much as possible and to avoid custom data-sources. This strategy will allows us to switch back-ends quickly if necessary.

Cheers,
Frank

12: Re: ]project-open[: Current status about AJAX, Sencha and AOLserver (response to 11)

Posted by Taras Mitran on 03/04/13 12:53 AM

Hey very nice. Seems like similar experiences as we've been doing our server-calls RESTful style too, and came up with tricks for sencha performance (gzipping/concatenation/minification, lazy rendering, buffered loading), and still supporting several openacs-based apps with newer interfaces.

One interesting thing I did reduced large amounts of reliance on tcl for an 'excel-like' interface that required all of the logic in the browser (for offline support). It was for what-if scenarios in the human resources work was write my own library for the client and re-use it on the backend in a plv8 function (javascript inside postgres). Openacs did nothing except handle the request and call the database function (after error checking, permissions). It allowed us just to 'replay' the front end actions in the database to keep the server in sync.

It would be interesting to have a chat one day and share techniques, especially relating to sencha, and patterns for highly dynamic apps. We could share a few things we've done that might have crossover to your apps, and vise versa (your ticket tracking app sounds cool).

Taras

13: Re: ]project-open[: Current status about AJAX, Sencha and AOLserver (response to 12)

Posted by Frank Bergmann on 03/04/13 09:09 AM

Hi Taras,

gzipping/...

Rrright, we've also added gzipping and a way to select the columns returned in object lists in order to reduce data volume, but I'm interested to learn about the other ones.

I'll send you a private message in order to schedule a call.

what-if scenarios

_Very_ interesting, we will have to work with what-if scenarios for project resource planning at some point in the future...

share

Here is a quick way to look at our Sencha client-side stuff. It's a complete ticket tracker/helpdesk system written in Sencha:
cvs -d :pserver:mailto:anonymous@cvs.project-open.net:/home/cvsroot checkout intranet-sencha-ticket-tracker

Cheers,
Frank