Forum OpenACS Q&A: webmail thoughts

Collapse
Posted by Mat Kovach on
In talking with a few people today (all day teleconfernces lead to
interesting conversations) the question of webmail in OpenACS came up
today.

Now I haven't really looked into the OpenACS/ACS webmail modules, but
for what I've seen on the list (and the lack of questions about it)
and the comments on the OpenACS 4.x Porting about it, it seems webmail
is not a particulary useful tool right now.

My question/comment/thought is, is it something that really needs to
be looked at (i.e: does anybody really want it?).

I like the idea of having webmail involved intergrated with OpenACS,
but I'm wondering about the real need/usefulness of such a feature.

I really don't like the ACS java solution for it (with a TCL software
based, I hate adding in another language).  Also, it ties to much to
the database which makes portability a real bear.

If it is determined that having webmail is a needed, and since it
appears to me that re-writing it is needed.  If that is done I'm
thinking a few things would have to be looked at:

Instead of delivering the mail to the OpenACS box (and database) would
it just be better to write a more conventional webmail interface
(using POP3/IMAP, possibly adding SSL Enabled pop3/imap for ISP that
support it), maybe offering the possibility of saving messages in the
database?

Using the filesystem and using virtual hosting then adding mbox or
maildir support to OpenACS, with a webmail interface on top of that?

Writing/using existing TCL code to parse a mail message, stuff it in
the database, and then having a webmail interface using that.  While I
think this is an interesting idea, this would be really hard on a high
volume site.

Possibly a combination (or all) of the above?

Just throwing this out there (and okay, I manage the SMTP
infrastruture for about 100,000 people, so I'm always looking for new
ways to present mail access to people on a more controlled field).

Thoughts/comments ?

Collapse
Posted by Malte Sussdorff on
A few students of mine have been thinking about improvements for ACES to use at our university. As we are not allowed to run our own mailserver they are going to investigate tying webmail to the POP Mailserver which runs at our university department. The idea is to POP the mail from the mailserver everytime the user logs into the webmail client. There are still some issues though like how do you get the mail from the webmail system to your offline Mailreader (has anyone written a POP server in TCL ?). Will keep you posted if and how they succeed.
Collapse
Posted by good bye on
here is the architecture I have thought up:

you want 2 physical machines , a mail server and a web server.
this can be done on 1 machine, but if you have more than
just a handful of users, you are going to want to have a dedicated
mailserver.

mail server:

qmail mail server

courier imap

web server:

your openACS installation +

a module which provides for webmail user administration

either machine: you want some AOL TCL scripts acting as
an imap client. i suggest writing these in a manner so
that they don't necessarily rely upon a bunch of ACS procs.
this way, you could even have a separate physical machine
somewhere (mail.yourdomain.com) that only acts as
an imap client for your users.

There are some links on qmail.org to hacks which
allow a qmail installation to authenticate with a postgres
database. I think that if you have the proper postgres
client libraries on the email server, you can do this
authentication between two machines. This would provide
for authentication of the mail users against the openACS
users table.

Collapse
Posted by Mat Kovach on
The only problem using using qmail with a postgres patch and authenticating against the OpenACS database is that you either have to modify the postgres patch (and require people to patch qmail) to use the OpenACS tables and create another table (with most likely duplicate information) in OpenACS.  Then if the person would rather use EXIM, Sendmail, or Postfix, they are a bit out of luck.  You would also have to authenticate using POP3 and IMAP, they you need to patch those utilities to also use postgres.

My thoughts is that for webmail is that we would want to be a mail server independant as possible and if OpenACS is going to use used for authentication, they we would want the mail controlled by OpenACS instead of attempting to share data out of the database to other servers and potentionally other servers.

Collapse
Posted by Michael Feldstein on
I use a service called POP3Now!, at http://www.pop3now.com. It
basically gives me a webmail interface for my existing email
accounts. I haven't a clue about how it works, other than to say
that it can gather email from multiple accounts without actually
removing the email from those accounts, so I can fetch the same
messages with my mail client of choice when I'm able. It's not a
perfect solution but it does what I need it to do.
Collapse
Posted by Ola Hansson on
I can make available a simple bash script I wrote that will install qmail, ucspi-tcp, and daemontools in under 5 minutes if anyone would like to try it.
There isn't any documentation besides trivial command line usage info and the source though.

Let me know if it can be useful so I can clean it up a bit...

Collapse
Posted by Ben Koot on
Ola, As we are about to start looking at the webmail options, I would be interested in your solution. Ben koot... chief non techno experimental OpenACS user, and moderator initiator of Klessebes in Holland. By the way greetings to all of you from our lovely tulip country a nd thanks for the great suport so far, also on behalf of Hans, our chief nerd, and Renny, our senior business advisor.
Collapse
Posted by Ola Hansson on
You can get a copy of the qmailinstaller from my site. (Please bear in mind this is my first Bash script ever, and that I'm not very proud of the code.)
Collapse
Posted by Mat Kovach on
Some very interesting ideas in there.  I think that it might be best to start off by creating interface for displaying messages, controlling who can use it, controlling how people use it etc.

Maybe at first a POP3/IMAP interface could be placed underneth that interface.  Then work out a SMTP->DB interface to handle the messages.

It would be nice to intergrate OpenACS's user authentication with the virtual hosting of various mail servers OR allow the remote mail retrevial.

Okay, well looks like I should really start looking at some code.

Collapse
Posted by good bye on
If you just want a mail client script to run under AOLserver, that is
actually quite easy. In fact, if you search the aolserver mailing
archives, you will find some POP routines posted. I think the post
dates way back to 1999. I have them in a tar file somewhere, so
if someone else doesn't find them. I'll dig them up this weekend.

There is at least one TCL imap package, so one could just take
the imap routines from that, and fold them into an aolserver
library...and write a couple scripts to grab and receive mail. the
tricky parts about each of these is going to be properly displaying
MIME extensions...but that type of thing is mostly well defined
and will just require some programming.

Collapse
Posted by good bye on
Most people who just want a webmail solution do not want to
store mail in postgres, unless you are writing a specialized app
such as a knowledge management system which lets people
search over everyone's email to find the answer to a specific
problem. Something like this would be very valuable, it just  is
beyond the scope of what most people want when they think
about a 'webmail' system.

note, it isn't impossible to use a database as the backend to a
standard webmail system...in fact, AOL stores all of its user mail
in sybase. it comes in handy when you need to find out which of
your users are using dirty words like 'breast' and 'pants.'

Collapse
Posted by Yon Derek on
Micheal, POP3Now looks like something I've been looking for however their webpage doesn't answer all of my questions. Would you be willing to share your experiences with POP3Now?

My questions:

  • do they have filters aka. virtual folders, i.e. ability to move messages to different folders based on some criteria e.g., where does it come from, who sent it etc. Hotmail has very limited capabilities. POP3Now's webpage doesn't mention that.
  • can you turn off this big header at the top which shows in their screenshot (i.e., stock quotes and headlines)?
  • can you maintain "multiple personalities" i.e., setup different accounts, e.g., one personal, one for customer service of your company etc.
  • they assure "no ads" but have "advertise with us" program?
  • in general, how does it work. Is their server fast, any things that drive you crazy etc.? Is it designed to handle a lot of e-mails (e.g., for someone who subscribes to 10 mailing lists).
  • Anyone else is using this or similar service (either pop/imap account with great web front or just a web front to existing pop/imap account) willing to share experiences?

Collapse
Posted by Mat Kovach on
Actually the way that I'm currently thinking is developing a webmail interface, and basically setting up the expectations of "Get a list of all my mail", "Delete mail", "Store this mail here" etc.  Then develop the underlying processes to handle it.  In that manner it gives people the option of adding what every type of processes they want to read the mail.  I would suggesting looking at POP3/IMAP support first since there is quite a bit of code out there already that can be at the process layer.

I think this might be a good approach and give quite a bit of flexability, while being a good base for people to expand on give an interface to the mail display and allow intergration into the rest of the system.

With the current webmail on both 3.x and 4.x in a state of "flux" and I'm more than willing to start attacking the problem.  I would just like to start approaching the problem in a manner that people would find useful.

I also think have a good approach to offering webmail (which is becoming an increasing more desired feature from my perspective) I would like to move OpenACS's webmail to (hopefully) something that seems a bit more intergrated, functional, and easier to setup.

For the most part, I think people will already have mail servers and POP3/IMAP access and bring mail into the other features of OpenACS might be helpful in the "community marketing" thoughts that have been discussed lately.

It seems that this thread has been helpful in opening the dicussion about webmail:  what people want, approaches, ideas.  Now we just have to continue to go forward and hopefully develop something useful.

Collapse
Posted by Michael Feldstein on

Yon, I'll try to answer your questions, although since I don't use the service heavily, my answers are going to be a bit tentative.<.p>

do they have filters aka. virtual folders, i.e. ability to move messages to different folders based on some criteria e.g., where does it come from, who sent it etc. Hotmail has very limited capabilities. POP3Now's webpage doesn't mention that.

Nope. It's pretty bare-bones. I wouldn't use it as my primary email interface; I just use it to get access to my email when I'm someplace where can't dial in to my SMTP server.

can you turn off this big header at the top which shows in their screenshot (i.e., stock quotes and headlines)?

Yeah, I'm pretty sure you can toggle that in the preferences page.

can you maintain "multiple personalities" i.e., setup different accounts, e.g., one personal, one for customer service of your company etc.

If you pay for their higher-level service, you can pull emails from multiple POP3 accounts. Email from each account is displayed separately from the other accounts. You can choose to email out through any of your POP3 accounts. I'm not sure about .sigs and the like.

they assure "no ads" but have "advertise with us" program?

Whether or not you see ads depends on the level of service that you pay for. The lowest level (which costs a couple of bucks a year for access to one POP3 account, last time a checked) displays ads.

in general, how does it work. Is their server fast, any things that drive you crazy etc.? Is it designed to handle a lot of e-mails (e.g., for someone who subscribes to 10 mailing lists).

As I said, I don't use it heavily, so YMMV. I don't think it would work terribly well for somebody who gets massive volumes of email every day because it doesn't really let you sort that email. On the other hand, it just looks into your server to see what you have on it at the moment, so if you go in with your regular mail client and clean out the server on a regular basis, you might be OK. The interface is fine, if bare-bones. Server can be a little slow at times, but not terrible.

Hope this helps.

Collapse
Posted by Talli Somekh on
One of the things the ACS and OpenACS community has always seemed to do is try and build a package in the current system rather than use best of breed applications from other environments.

I don't think that this is all bad most of the time (see content management) but I think that for something as complex as webmail it may be easier to try and integrate one of the other systems with OpenACS.

We are currently using Squirrell Mail (http://www.squirrellmail.org). It's a little buggy and not entirely mature. We are going to try IMP (http://www.horder.org/imp) as a substitute. Both are built in PHP and use Apache.

I am admittedly not qualified to speak about how hard it would be to mix these systems. I will only say that it seems like a real bitch to build a good webmail system from scratch, even with the headstart that OpenACS gives.

Mat, I know you said you hate integrating two languages, but would it all be easy to integrate systems that solve your problems but aren't in the same language?

talli

Collapse
Posted by Jonathan Ellis on
IMP is at horde.org, not horder.org.

I actually tried to install this under PHP/AOLServer a while ago but the only conclusion the Horde developpers and I came to was, "It didn't work."  Somewhere along the line its request to the imap server wasn't reaching it, and I didn't have time then to learn PHP and debug their code.

Collapse
Posted by Mat Kovach on
If you are going to use something like IMP, TWIG, etc. then you would be much better off using and apache server and using user authentication from the OpenACS database.  (At least I think).  There are all good products, but my hope is to have something intergrated with OpenACS (Use the calendars, bookmarks, etc.) along with webmail.

I'm looking right now to see how much of the current webmail interface can be pulled out of both 3 and 4 and how much it would take to retrofit POP3/IMAP support (to begin with) into it.  There are some respectable TCL libs out there we can use for the networking, so really I think just an inteface would be a good start.

In the next day or so I'll write up a design spec.  Since there is already a webmail interface and there really isn't a current tcl web interface, a simple interface will not be that hard to put together.  Heck, I'm even willing to do the work, I just would like to make sure I'm going along a path that others will find useful.

Collapse
Posted by Andrew Piskorski on
Has there been any further progress on AOLserver/OpenACS webmail clients?

In particular, I would like to find a near-term solution that:

  • Is simple, works right now, or is fairly easy for me to glue together into something reliable.
  • Runs in AOLserver, without needing a PHP module or anything else that's currently unstable or overly complicated.
  • Supports IMAP.
  • Can use SSL to encrypt the link to the IMAP server.
  • Here are some features that I personally don't care at all about right now, but of course other folks will want in the future:

    • Integration with OpenACS user authentication.
    • MIME support.
    • Both IMAP and POP3 support.
    • Highly efficient, massively scaleable.
    • A webmail solution is not too high a priority for me, which is why I emphasize the "simple, not a lot of work" requirements. But a basic solution would be very nice. And if there's a simple solution who's basic design would allow me to later work on it and turn it into a full webmail solution for use by the community, that would be ideal. Anyone have suggestions or advice?

Collapse
Posted by David Walker on
I am working on a C module to display MIME parts from a file or the database and then combining it with a qmail maildirs configuration and some integration between qmail and the OpenACS user database and theoretically have a robust, high performance, fully OpenACS integrated webmail solution.

This would mean ACS and the mail server would run on the same machine or at least share a file system but it would allow you to use any existing qmail solutions for pop, imap, and smtp.

I can already read and display mime parts from a stored message using the reformime utility from the sqwebmail package and the ns_returnbinary command from my nsbinarysupport module. (it is at http://www.vorteon.com/download/)
Collapse
Posted by Ben Koot on
Is there any progress in adding a releable webmail service to ACS?
Collapse
Posted by MaineBob OConnor on
Hi Ben,

Wow, I started reading this thread and didn't realize until the end that it is over a year old!

I just started a new thread about my bulk mail experiences.

https://openacs.org/bboard/q-and-a-fetch-msg.tcl?msg_id=0005N0

-Bob

Collapse
Posted by Rafael Calvo on
Two of my students are just starting to work on this. THey should have a neat webmail working by the end of september. I have asked to post the design documents ASAP.

cheers

Collapse
Posted by Malte Sussdorff on
Though I'm still against dublicating the efforts for a webmail system (there are really good ones out there that are developed with a significant amount of effort) itself, I am pleased that your students are going to work on it.

They might take a look at IMP 3.0, which is the webmail system we are using for 30000 people and they are very pleased so far. And yes, we do have integration into ACS working, with tools to signup users to the webmail system and other integration work (isn't HTTP cool :-)).

If you want surely Venkatesh can give your students a demo otherwise look at http://www.horde.org/imp/3.1/

Collapse
Posted by David Geilhufe on
Malte-

You mention that you have integration working? Was this some super customized effort? How would I duplicate your results if I wanted to?

The reason I ask, is that there are a bunch of mature OSS products that would be greate to integrate into OACS without having to rewrite them. Would the appropriate strategy be to do a default install of a webmail program, IMP for example, and then install an OACS package that is the "IMP integration" package?

Or is this impossible due to changes required in IMP?

The goal I have in mind is not to prove integration possible, which it clearly is, but to make integration as close to plug and play as possible.

Collapse
Posted by Peter Marklund on
How would such a webmail be integrated with OpenACS? Would the emails be full-text searchable? I suppose one obvious but small advantage would be that you would have immediate access to all the email addresses of all the users on the site.

On the other hand, we have always encouraged using discussion forums rather than email for correspondance that can be of general interest. It happens very often that I see email exchanges that should really be in the Bboards. We need to be wary to not overuse email.

I rely on Fastmail.fm for email. FastMail.fm is the ultimate Hotmail killer. It is easily the best email service that I have seen (has POP and IMAP, virus and spam protection and loads of other features) and an extremely efficient and usable webmail interface. Fastmail also has a very active Discussion Forum of users and developers. In fact, Fastmail is so good that I wonder why anyone would go to the trouble of hosting their own webmail, unless there are some significant integration benefits that I don't know of.