Forum OpenACS Development: webmail status?

Hi,

Has anyone thought/started to port webmail?

From an IRC channel convo I had with Kapil, he mentioned a good way to solve the webmail problem might be to rewrite it in Python. He mentioned that much of the functionality and feature set needed (POP, SMTP, IMAP, etc) is built into the language (but don't ask me how).

Since webmail in the ACS is by all accounts a bit on the crappy side, Oracle based and written in Java, it might make sense to simply rebuild the thing, no?

Kapil, can you weigh in here?

talli

Note that PHP is not 100% stable on AOLserver yet.  There are still some thread-safety issues, at least the last time I looked at it.  How or when the PHP team is going to fix that is not known.

That said, I have managed to run PHP (and IMP as my webmail) on my home box.

--cro

To followup on cro's statement, it's been our experience that SM+PHP+AOLserver is not rock solid, but it is quite usable for a small group of users. We have been using it for about 6 months with good results for light use.

And, don't forget to install keepalive if you use PHP+AOLserver.

How large is large?

I would put a limit on message size if I were hosting webmail.  20
meg per message if I like the people, much smaller if they were
strangers.

While keepalive is necessary, there are other ways to add more stability, and that is to use a reverse proxy (SQUID, Apache, nsvhr) in front of your webservers and then split the python or php portion into a separate AOLserver instance.  If each AOLserver is using the same domain stem, and you cookie right, both servers should be able to see the session id cookies(, and should you cookie wrong, then a smop of IPC should get you a module that lets the two servers coordinate session ids.)

Then should your python/php server die, or when it dies, it will only take down the python/php server (until keepalive kick starts it), and leaves your other services running fine.

Side note: yes, I found the tcl-lib mime stuff wonderfully easy to use, and too slow to be of use.  Without profiling it, I just suspected the tcl interpreter based mime encoding was way slow compared to some honest op codes.

Does anyone have the ability to return binary data from aolserver to
the browser?  Wouldn't that be needed to return attachments without
saving them to disk first?

I heard of someone patching aolserver to return gzipped content.
Would that version be able to return other binary content as well?

have/will your uu??code functions make it into an official aolserver
release?  What are the chances of the modified ns_return making it
in?

I've submitted the patch (against 3.4.2) to aolserver via sourceforge, but i haven't heard anything from them yet (it's only been a couple days). It's available here.

responding to talli's request for comments.

i totally agree with john that the ad webmail implementation is rather broken (design) and inflexible. i haven't used phpwebmail, so i'll restrict my comments to python.

python has wonderful support for the mail protocols that comes with every python distribution in the python standard library and brings alot of additional functionality to aolserver from unittests to unicode to xml processing.

for example here's a session to check how many messages and how large my mailbox is using pop
import poplib
server = poplib.POP3('mail.wm.edu')
server.user('kvthan')
server.pass_('guess')
print server.stat()
<blockquote>> (10,12350)
</blockquote>
server.quit()

(you can try this @ home in any python interpreter shell)

documentation links
# pythons standard lib internet protocols
http://www.python.org/doc/current/lib/internet.html

# python std lib internet data handling
http://www.python.org/doc/current/lib/internet.html

basically the libraries take care of the protocol communication while exposing the protocol functions. its also comes with a mime library (to be updated in the standard lib by www.sf.net/projects/mimelib) and is used by one of the most widely used mailing list softwares, mailman (www.list.org)

as for mime-encoding speed. i agree its important, but i think imap serves to mitigate a large subset of mime-parsing, by preparsing entities into envelopes and message bodies. i think really think webmail should focus on supporting imap not pop, since while pop can be used in a poor man's imap mode, it really isn't designed for this and might be problematic for large boxes/messages, concurrency due to mail acct locks, security.

i've been working with python in aolserver (http://pywx.idyll.org), and i've found it to be remarkably stable. i've stress tested it, done wierd and wacky things like embedding zope in aolserver, and my dev server has been up for over a month with a stable footprint. i haven't yet done much interaction with openacs but after wrassling with the request processor i expect it will be straightforward (hopefully). i'm not sure about the php integration for aolserver, but ns_python can execute tcl commands and is pretty much a full wrapping of the aolserver c api minus some of the socket and callback functionality (python has its own socket library). the ability to execute tcl functionality means some of the session handling functionality and openacs api can be reused by a python based package.

i don't think the picture is entirely rosy though, one problem i've run into is finding a good gpl templating package for python, to date i've mainly used the ones from zope (DTML, and Zope Page Templates), but they aren't compatible with the GPL (yet). ns_python can be used from adp, but adp on a whole feels clunky compared to the relative separation of logic and presentation offered by the acs templating system.

probably the larger problem is acceptance in the openacs community. the question is are people willing to accept a package that requires the installation of the python aolserver bindings?

Minimally, what would it mean for a webmail package to be
"integrated with the datamodel"? What parts would benefit from
integration? Is it permissions? Do you want to stuff the emails
themselves into the content repository? Is it possible to simply
build some kind of a bridge tool that would make it easy for
people to integrate their webmail package of choice with the right
pieces of OpenACS?

I've created a module that makes the commands ns_returnbinary and ns_writebinary for AOLServer. It can be downloaded at http://www.vorteon.com/download/. (Thanks Vinod, I couldn't have accomplished this without your code to learn from)

I've had some success with returning images and binary files out of mime encoded message files using the reformime utility from sqwebmail and the tcl exec command with ns_returnbinary and ns_writebinary. I'm still working on doing the same thing from C. (I'm new to C though. This could take a while)

I think webmail should store messages in the file system but store information about mime sections/ attachments in the database along with the text/plain or text/html section for easy searching.

I'd like to see the system coexist with pop and imap as well and I think the maildir storage method would cover all my requirements.