Forum OpenACS Development: imap4 for ACS Mail Lite incoming email

Collapse
1: imap4 for ACS Mail Lite incoming email
Posted by Benjamin Brink on 07/02/17 09:18 AM
Hi all,

A current project needs to integrate a custom email system with ACS Mail Lite's incoming email feature.

The road to hacking the suggested implementation seems steep, and fragile to any changes in the underlying component versions.

I'm going to try adding imap4 as a general alternate. It might scale better by allowing the imap server to be on another system, just as outgoing email has SMTP.

There are at least two IMAP4 APIs available:

imap4 via tcllib: https://core.tcl.tk/tcllib/doc/trunk/embedded/www/tcllib/files/modules/imap4/imap4.html

NaviServer's nsimap: https://bitbucket.org/naviserver/nsimap/src

Any reason why one should be used over the other?

nsimap seems the natural choice..

Any suggestions on integrating with acs-mail-lite?

thanks,
Ben

Collapse
2: Re: imap4 for ACS Mail Lite incoming email (response to 1)
Posted by Gustaf Neumann on 07/02/17 02:34 PM
We are not using incoming mails on any of our OpenACS systems. There is probably no out-of-the box solution for a more general handling than described already in https://openacs.org/xowiki/incoming_email

Performance- and feature-wise, i would expect nsimap to be quite good. It was implemented by Vlad as a relative thin module based on the UW IMAP c-client library from washington.edu, which is used in many imap clients and other scripting languages. It is actually the reference implementation for IMAP (see [1]). Feature-wise, this is very complete [2]. To remove some hard edges, I have just updated the module to compile on recent systems. However, as nsimap is not used widely, expect a few hick-ups when using this. The code compiles as well with panda-imap [3].

One other advantage is that the NaviServer code scales in general better than Tcl since it does not run into the problems like e.g. the 1024 open file descriptors. There is some work done in the Tcl community to overcome this, but it will take a while until it hits mainstream.

One further option is to use NaviServer nssmtpd module, which allows to act as a SMTP deamon. One of my students did some work last year to add STARTTLS support to it.

In general, when using (Tcl) client packages, one problem is a scalable integration, which might worry people or not. When e.g. a client (e.g. a Tcl function) is called within a connection thread, it might occupy the connection quite a long time. Therefore, the server might run out of connection threads, when e.g. a high number of clients is doing this at the same time. Therefore, it is a goal to move out such operations into the background and handle this in a multi threaded fashion. This is more feasible in C than in Tcl, especially when code is depends on the state of a certain Tcl interpreter. The nsimap code supports multiple sessions, so i would expect advantages in this respect.

-gn
[1] https://en.wikipedia.org/wiki/UW_IMAP
[2] https://www.washington.edu/imap/IMAP-FAQs/
[3] https://github.com/jonabbey/panda-imap

Collapse
3: Re: imap4 for ACS Mail Lite incoming email (response to 2)
Posted by Benjamin Brink on 07/02/17 05:57 PM
Thank you, Gustaf.

ns_cheers 1025

Ben

Collapse
4: Re: imap4 for ACS Mail Lite incoming email (response to 2)
Posted by Malte Sussdorff on 07/09/17 06:57 AM
I wrote an IMAP integration with the nsimap client for ]project-open[ some time ago.

https://gitlab.com/cognovis/intranet-mail has the code. Maybe this helps, the project was somewhat abandoned though I probably start working on it again in October as we need to include (and log) incoming reply-to E-Mails in for a customer.

Collapse
5: Re: imap4 for ACS Mail Lite incoming email (response to 4)
Posted by Benjamin Brink on 07/11/17 03:03 AM
Hi Malte,

Thank you for offering the code example. Examples are handy.

At this point, my obstacle is in understanding how OpenACS callbacks work.

I understand the theory. Callbacks register events. And contracts are triggered by events. And I'm familiar with even driven programming. And yet, in my own typical way, I'm unable to yet make sense of how it's implemented in OpenACS. I'm relying heavily on existing callback cases in OpenACS, and document you wrote for Incoming email https://openacs.org/xowiki/incoming_email to help sort it out. Thank you for that documentation.

cheers,
Ben

Collapse
7: Re: imap4 for ACS Mail Lite incoming email (response to 5)
Posted by Antonio Pisano on 07/11/17 03:44 PM
Hoping to clarify about callbacks, even if in a very non-scientific way: one defines contracts for functions in package A, then packages B, C, D... can all define implementations of such contracts that will all fire in theoretically unspecified order (but alphabetical in practice) whenever one calls:

callback the::contract::name -args $args

Basically the system will loop through every registered implementation and fire it one at a time, lappending each result to the final return value of the callback call.

This I guess you have seen already, but for reference:
https://openacs.org/api-doc/proc-view?proc=callback&source_p=1

Collapse
8: Re: imap4 for ACS Mail Lite incoming email (response to 7)
Posted by Benjamin Brink on 07/11/17 07:31 PM
Hi Antonio,

Thank you for the explanation. Indeed. The main point of confusion for me has been the concept that a callback collects outputs of contracts instead of each contract terminating as a separate thread.

I was expecting a callback to execute contracts independently via new threads.

Fully sequential execution within the same thread seems prone to having a contract adversely affect execution of contracts that follow (in same thread), since contracts are made and operate outside of the callback environment.

This arrangement has a variety of implications. Consequences include a contract possibly affecting input parameters or having a procedure susceptible to timeout conditions that delay processing of some contracts beyond a scheduled callback's expected execution time --when the callback is triggered as part of a scheduled proc.

Designing callbacks (and contracts) that work within this environment is critical for a centralized service, such as ACS Mail Lite.

Questions need answered in the design of a callback: Should a callback 'learn' to ignore a contract with a history of taking 'too long' to process or when a contract errors a certain number of times?

Now that the confusion is cleared, I'm set to resolving the challenges.

cheers,
Ben