Forum OpenACS Development: webmail status?

Collapse
Posted by Rafael Calvo on
Hi,

Has anyone thought/started to port webmail?

Collapse
Posted by Don Baccus on
No.  There are a few problems with it:
<ul><li>It is thoroughly Oracle/SQLJ'd with no pretense of portability.
<li>It's not integrated with acs-mail or acs-messaging so there's duplication of effort.
<li>I've not heard many good things about the UI but I've not used it myself.
</ul>
<p>A good first step would be to get someone to agree to fire it up under Oracle and do an evaluation of the UI.  If it's good enough to keep we could consider ripping out the guts of the datamodel and all the SQLJ code and look into doing an acs-mail based variant.
.
Collapse
Posted by Talli Somekh on
From an IRC channel convo I had with Kapil, he mentioned a good way to solve the webmail problem might be to rewrite it in Python. He mentioned that much of the functionality and feature set needed (POP, SMTP, IMAP, etc) is built into the language (but don't ask me how).

Since webmail in the ACS is by all accounts a bit on the crappy side, Oracle based and written in Java, it might make sense to simply rebuild the thing, no?

Kapil, can you weigh in here?

talli

Collapse
Posted by Michael Feldstein on
FWIW, I believe OpenForce has documented how to integrate
SquirrelMail, a PHP-based system for IMAP, with OpenACS:

http://openforce.net/library/doc/openacs/webmail-with-php-on-ao
lserver.adp

Collapse
Posted by C. R. Oldham on
Note that PHP is not 100% stable on AOLserver yet.  There are still some thread-safety issues, at least the last time I looked at it.  How or when the PHP team is going to fix that is not known.

That said, I have managed to run PHP (and IMP as my webmail) on my home box.

--cro

Collapse
Posted by Jon Griffin on
I am running SquirrelMail,Qmail and courier IMAP.

WebMail is awful. Everything stored in Oracle and users can't POP or IMAP. My feeling is that it is a complete waste of time to port. Many, MUCH better alternatives exist and at some point we have to realize not everything is a good fit for OpenACS.

Collapse
Posted by Arjun Sanyal on
To followup on cro's statement, it's been our experience that SM+PHP+AOLserver is not rock solid, but it is quite usable for a small group of users. We have been using it for about 6 months with good results for light use.

And, don't forget to install keepalive if you use PHP+AOLserver.

Collapse
Posted by Dan Wickstrom on
What's really needed to make webmail viable under aolserver is a good mime handling library similar to those that are implemented in java - something like a ns_mime aolserver module.  I've used the mime package in tcllib, and it's just to slow for handling emails with large attachments.  It's use in webmail would easily open you up to a major DOS attack.
Collapse
Posted by David Walker on
How large is large?

I would put a limit on message size if I were hosting webmail.  20
meg per message if I like the people, much smaller if they were
strangers.

Collapse
Posted by Don Baccus on
I wonder where the time's eaten up?  Vinod got something like four orders of magnitude performance with ns_uuencode over Tcl code he poached from some library somewhere (he had to fix ns_uuencode first).

It's possible that unencoding is expending a lot of time, too, which could be fixed by hacking up an ns_uudecode.

Of course, I also kinda take Jon's comment to heart.

While keepalive is necessary, there are other ways to add more stability, and that is to use a reverse proxy (SQUID, Apache, nsvhr) in front of your webservers and then split the python or php portion into a separate AOLserver instance.  If each AOLserver is using the same domain stem, and you cookie right, both servers should be able to see the session id cookies(, and should you cookie wrong, then a smop of IPC should get you a module that lets the two servers coordinate session ids.)

Then should your python/php server die, or when it dies, it will only take down the python/php server (until keepalive kick starts it), and leaves your other services running fine.

Side note: yes, I found the tcl-lib mime stuff wonderfully easy to use, and too slow to be of use.  Without profiling it, I just suspected the tcl interpreter based mime encoding was way slow compared to some honest op codes.

Collapse
Posted by Don Baccus on
I think - but I'm not 100% certain - that tcl-lib is where Vinod got the uuencoder.  I forget how much faster the ns_uuencode routine was once he  got it fixed (UTF8 broke it for Tcl8) - I think four orders of magnitude is an UNDERstatement...
Collapse
Posted by David Walker on
Does anyone have the ability to return binary data from aolserver to
the browser?  Wouldn't that be needed to return attachments without
saving them to disk first?

I heard of someone patching aolserver to return gzipped content.
Would that version be able to return other binary content as well?

Collapse
Posted by Vinod Kurup on
I think - but I'm not 100% certain - that tcl-lib is where Vinod got the uuencoder. I forget how much faster the ns_uuencode routine was once he got it fixed (UTF8 broke it for Tcl8) - I think four orders of magnitude is an UNDERstatement...

Yeah, the tcl code that I'm using is from tcllib. Unfortunately, I made a mistake when I tested the code initially. So, while I told Don that the fixed ns_uuencode was MUCH, MUCH faster than tcllib, it's actually only about 60 times faster. (Sorry, Don!). Most of the time is spent while splitting the string into 48-byte pieces to be sent to ns_uuencode. When I move that task inside ns_uuencode so that you can send arbitrary length strings to ns_uuencode, then ns_uuencode is about 2000 times faster.

Does anyone have the ability to return binary data from aolserver to the browser?

I think this requires that ns_return and ns_write be rewritten as Tcl_ObjCmdProcs. This is what I did for ns_uuencode, and it wouldn't be too hard to make this change for those procs as well (I think).

Collapse
Posted by David Walker on
have/will your uu??code functions make it into an official aolserver
release?  What are the chances of the modified ns_return making it
in?
Collapse
Posted by David Walker on
and, Vinod, can I get a copy of your patch for ns_uuencode?
Collapse
Posted by Vinod Kurup on
I've submitted the patch (against 3.4.2) to aolserver via sourceforge, but i haven't heard anything from them yet (it's only been a couple days). It's available here.
Collapse
Posted by David Walker on
Can I get a copy of your 3.3ad+13 patch as well?
Collapse
Posted by kapil thangavelu on
responding to talli's request for comments.

i totally agree with john that the ad webmail implementation is rather broken (design) and inflexible. i haven't used phpwebmail, so i'll restrict my comments to python.

python has wonderful support for the mail protocols that comes with every python distribution in the python standard library and brings alot of additional functionality to aolserver from unittests to unicode to xml processing.

for example here's a session to check how many messages and how large my mailbox is using pop
import poplib
server = poplib.POP3('mail.wm.edu')
server.user('kvthan')
server.pass_('guess')
print server.stat()
<blockquote>> (10,12350)
</blockquote>
server.quit()

(you can try this @ home in any python interpreter shell)

documentation links
# pythons standard lib internet protocols
http://www.python.org/doc/current/lib/internet.html

# python std lib internet data handling
http://www.python.org/doc/current/lib/internet.html

basically the libraries take care of the protocol communication while exposing the protocol functions. its also comes with a mime library (to be updated in the standard lib by www.sf.net/projects/mimelib) and is used by one of the most widely used mailing list softwares, mailman (www.list.org)

as for mime-encoding speed. i agree its important, but i think imap serves to mitigate a large subset of mime-parsing, by preparsing entities into envelopes and message bodies. i think really think webmail should focus on supporting imap not pop, since while pop can be used in a poor man's imap mode, it really isn't designed for this and might be problematic for large boxes/messages, concurrency due to mail acct locks, security.

i've been working with python in aolserver (http://pywx.idyll.org), and i've found it to be remarkably stable. i've stress tested it, done wierd and wacky things like embedding zope in aolserver, and my dev server has been up for over a month with a stable footprint. i haven't yet done much interaction with openacs but after wrassling with the request processor i expect it will be straightforward (hopefully). i'm not sure about the php integration for aolserver, but ns_python can execute tcl commands and is pretty much a full wrapping of the aolserver c api minus some of the socket and callback functionality (python has its own socket library). the ability to execute tcl functionality means some of the session handling functionality and openacs api can be reused by a python based package.

i don't think the picture is entirely rosy though, one problem i've run into is finding a good gpl templating package for python, to date i've mainly used the ones from zope (DTML, and Zope Page Templates), but they aren't compatible with the GPL (yet). ns_python can be used from adp, but adp on a whole feels clunky compared to the relative separation of logic and presentation offered by the acs templating system.

probably the larger problem is acceptance in the openacs community. the question is are people willing to accept a package that requires the installation of the python aolserver bindings?

Collapse
Posted by Don Baccus on
Well ... requiring Python's not nearly as bad as requring Oracle SQLJ, that's for sure!

My major concern is complexity.  Python's a great language.  At the moment to understand everything in an OpenACS 4 release you need to be able to read: Perl, SQL, Tcl, the templating system/.adp. That's one language less than ACS 4 Classic (which tossed Java into the mix).  Perl's only used for a couple of external scripts, so you can heavily customize an OpenACS 4 installation without knowing it.  But a Python webmail package would require one to know it if you were to want to customize it.  Also ... not only do you have the problem of there being no good GPL'd template system but you'd be forcing folks to learn *two* templating systems to effectively customize the look-and-feel of their site.

So I'm not thrilled with the notion of raising the bar in this way.

On the other hand, integration with the OpenACS 4 datamodel would certainly make it more attractive than an independent, third-party webmail package...

Collapse
Posted by Michael Feldstein on
Minimally, what would it mean for a webmail package to be
"integrated with the datamodel"? What parts would benefit from
integration? Is it permissions? Do you want to stuff the emails
themselves into the content repository? Is it possible to simply
build some kind of a bridge tool that would make it easy for
people to integrate their webmail package of choice with the right
pieces of OpenACS?
Collapse
Posted by Don Baccus on
User administration, subsiting ... Log on once and be able to visit the entire site, for instance, rather than requiring log-in to a third-party  webmail package.  Subsiting so that webmail could be appropriately placed in the site map (and permissioned etc).  Proper navbars generated  for webmail mounted in various places in the map.  Integrated "look and feel" (OK, that's not a datamodel integration issue, but a consequence of using a single templating system).

Stuff like that.

Collapse
Posted by David Walker on
I've created a module that makes the commands ns_returnbinary and ns_writebinary for AOLServer. It can be downloaded at http://www.vorteon.com/download/. (Thanks Vinod, I couldn't have accomplished this without your code to learn from)

I've had some success with returning images and binary files out of mime encoded message files using the reformime utility from sqwebmail and the tcl exec command with ns_returnbinary and ns_writebinary. I'm still working on doing the same thing from C. (I'm new to C though. This could take a while)

I think webmail should store messages in the file system but store information about mime sections/ attachments in the database along with the text/plain or text/html section for easy searching.

I'd like to see the system coexist with pop and imap as well and I think the maildir storage method would cover all my requirements.
Collapse
Posted by kapil thangavelu on
i've done some further investigating today of the use of python with the openacs. my "wrangling" with the request processor was over fairly quickly (at least for simple interaction).

don, i agree that an integrated solution would be a much better approach, esp. one that unifies the template handling. as for complexity , i feel that programming imap w/ python would be much simpler and more maintainable than the alternatives.

in effort to solve the template issue, i did some more investigation of the ats (and the upvar hell that it is) and was able to integrate python with the ats (with a little more rp work). so python a based solution can also share the same templates that will be used throughout a site:)

i'd like to do some more testing of persistent imap connections and load testing, but i think that a python based webmail is definitely feasible. i don't think its going to nesc. be a single logon system, as most people don't keep won't have their acs passwords the same as their email accnt passwds, and afaik acs passwords aren't stored clear text.

i seriously dislike the approaches of using storing a users email on the server, mail systems like imap were designed for this type of usage. i don't see any reason to reinvent the wheel, by purposefully burdening the openacs with the complexities and overhead of user mail management and the implied restrictions on user access.