Forum OpenACS Development: refactoring acs-mail-lite for use with imap

Hi all,

I've reached some complexity in refactoring acs-mail-lite, and would like your input on how to approach resolving.

Currently, notifications and acs-mail-lite packages create headers that are used to evaluate replies.

The reply headers can be used to:

1. validate than an email fits a pattern consistent with replies from outbound email.

2. return an object_id and type_id for use with callbacks.

At issue:

1. Although replies might fit a profile, they are not checked for a unique reference. This makes the process of faking a reply somewhat trivial. Also acs_mail_lite::valid_signature isn't used.

2. Internal references including object_id and type_id are sent in emails.

Does it make sense to pass a randomized external reference, much like a secure session cookie paradigm instead?

If so, I've created these procs for this purpose:

Details here:

Does it make sense to create an InboundEmailMode parameter in acs-mail-lite (or notifications), which would allow switching between existing paradigms and other paradigms, such as the IMAP one?

The parameter would be referenced by notifications package as well as acs-mail-lite. This means changes to notifications package will be submitted with revisions to acs_mail_lite.

If so, a switch would be added to existing message_id generators to plug a few anticipated vulnerabilities that would be opened up from integrating email replies into OpenACS packages.

Otherwise, it seems a proc that collects input from inbound email would have to check at least three different message_id types generated by Openacs in order authenticate --at least that the message_id is in proper form.

Your thoughts?


Posted by Benjamin Brink on
I think the following will work.

acs-mail-lite is the boundary for processing inbound and outbound email. Outbound notifications and acs_send_mail send mail via acs_send_mail::send. All inbound are handled via acs_send_mail.

Intercepting outbound message_id and replacing with the new paradigm will work for new cases.

Other implementations can then continue unchanged thereby limiting risk for affecting any existing deployment.

Posted by Gustaf Neumann on
Concerning faking replies etc. The proper mechanism to avoid tampering with object_ids are signed variables [1]; it should not be necessary, to invent new mechanisms.


Posted by Benjamin Brink on
Thank you, Gustaf,

I'm definitely interested in re-using code.

export_vars uses ns_set, which doesn't persist after a server restart. A reply may be expected to work for a period of time after a few restarts --at least for the deployments I'm considering using this in.

ad_set_signed_cookie would work if it could be used outside of cookies. And yet, it uses ns_sha1 which appears to be twice as slow as the work in progress (WIP) solution.

The WIP creates a unique id using ns_base64encode of a randomized big_int. Granted, the draft currently saves immediately to the database, which is much slower. The database write could be worked into a separate scheduled thread that batches values saved via ns_nsv similar to export_vars.

Is there another way that should be considered?


Posted by Gustaf Neumann on
Signed variables certainly work with restarting the server, it does not depend on the ns_sets, used for its construction. The ns_sets are just for temporary usage, like other local variables. Btw, the life-span of most ns_sets are per-request. Signed variables would be completely useless, if they could not be used across requests. You can specify e.g. the span of validity for a signed variable.
Posted by Benjamin Brink on
Thank you, Gustaf.

I should use it like this:

export_vars -sign -url <message_id> var1 var2 var3

And yet, how to retrieve?

As far as I can tell, to use ad_verify_signature, the data needs to be embedded in the email. And yet, the point is to *not* expose or publish the data external to the system.

Also, this grep doesn't find any examples besides docs to get hints from:
packages# grep -R " -sign" *

Posted by Gustaf Neumann on
OpenACS has multiple variants for signed value checking. The most basic one is the following, where ad_verify_signature_with_expr [1] returns either the expiration time or 0 if the validation fails
set value 123
set secret "secret phrase" 
set signature [ad_sign  -max_age 600 -secret $secret $value]
ad_verify_signature_with_expr -secret $secret $value $signature
Posted by Benjamin Brink on
I see how to use a signed variable for email now.

Where a url is supplied in an email for a user to get via a browser, standard export_vars -sign -url ... applies.

Where input requires authenticating a reply and obtaining associated form inputs, pass the uniqueID mapped to the inputs, where the uniqueID is signed using export_vars and adjusted to fit email message-id specs.

This keeps from leaking data, and message-id is re-generated using existing code.

Thank you, Gustaf!
That makes the implementation much cleaner.

Posted by Benjamin Brink on
Based on current feedback:

A randomized bigint does not provide a sufficient size for unqiueID with emails.

Using an sha1 hash of message_id would be sufficient.

'export_vars -sign' paradigm allows for any number of variables to be passed, and so should be a paradigm to copy.

Data is checked for change using ad_verify_signature. However, this process requires the data to be exposed potentially via email and shared via notifications.

Also, emails have external persistence. An external persistence of data limits the usefulness in environments practicing strict privacy and security policies.

One way around this is to map the hash to a local table that stores the name value pairs.

I'll make these procs:

acs_mail_lite::vars_export unique_id_hash var1 var2 ..varN that maps a unqiue_id_hash to variables and their values and stores in database.

acs_mail_lite::vars_import unique_id_hsah that returns a list of name value pairs from hash of message_id

This way, an industrious openacs-core developer can later re-implement a version of it in acs-tcl, or add it's functionality to export_vars.

I've implemented something like this in q-forms package for ecommerce apps, where name value pairs can be passed between form pages via the database only.

This way, the IMAP implmentation in acs_mail_lite will meet some strict requirements while providing minimal changes to existing acs-mail-lite code and not requiring changes in core external to acs-mail-lite.



Posted by Benjamin Brink on
Revised this to use export_vars -sign unique-id_hash, where unique_id_hash is message-id generated from existing code and mapped to internal data passed with email.. (See my prior post 9 in this thread).