Forum OpenACS Development: html in emails.

Collapse
Posted by Mohan Pakkurti on
hi!

I have been looking at how to support html in emails that are sent from OpenACS. Right now there is no recommended method to send MIME encoded emails. The general assumption/choice seems to be that we just don't send out email messages with html tags in them, and all the current OpenACS and .LRN packages reflect this. Wouldn't it be nice to support html in notifications, bulk-mail etc.

  • Should we look into supporting mime encoded emails form OpenACS?
  • How to provide a simple feature to send html in email messages?
My ideas is that we could add an optional parameter to acs_mail_lite::send. Packages that need to send html emails could then pass the html part in that parameter. The functionality to send mime messages could be supported by using tcllib - mime library. I have tried this using an implementation by Doug Harris and it works. This could be a simple start to support html emails.

Comments or ideas about this topic?

Collapse
Posted by David Kuczek on
Hello Mohan,

oacs 3.2.5 supported the option to send Emails in html... I can remember that I had to fix some bugs on the spam module to make html-emails work, but now it works. You could see how it worked in 3.2.5 and port it to 4.x

Also check out this thread:

https://openacs.org/bboard/q-and-a-fetch-msg.tcl?msg_id=0001oG

Collapse
Posted by defunct defunct on
Yes, its an excellent idea. Do we need to be careful about what HTML is allowed out?
Collapse
4: Re: html in emails. (response to 1)
Posted by Doug Harris on
The good thing about the HTML email proc that I created is that it uses the tcllib mime library so there's lots of flexibility there.

The bad thing about the HTML email proc that I created is that it uses the tcllib mime library and AOLserver doesn't play nicely with tcl packages and these libraries. Since you're using it, you probably have seen that it sources base64.tcl, mime.tcl, and smtp.tcl each time the proc is called. There's gotta be a better way.

Off the top of my head, I don't recall the issues that AOLserver has with tcllib and packages, but at the time that I wrote that code I researched here and on the AOLserver list archives and ended up with this solution. I would not recommend this if you're going to be pushing out a large volume of email.

Collapse
5: Re: html in emails. (response to 1)
Posted by Caroline Meeks on
Hi Doug,

Welcome and thanks for posting!

Thanks to your code and Mohan we have html email running on SloanSpace for a month or two and we haven't seen any problems yet.  We have medium volume, not sure how many emails but we have 9000 registered users, just under a 1000 users log on each day.  We use html for both notifications (forums and survey) and bulk-mail. Half the users complain about getting too much mail, the other half keep asking for more ways to send spam :) but the aolserver and tcllib hasn't given us any trouble.

Caroline

Collapse
6: Re: html in emails. (response to 1)
Posted by Richard Hamilton on
This evening I have been trying to get my openacs-4-6 to send email password reminders as html and was meeting with mysterious problems.

Having dissected the file line by line it became obvious that the problem was an unwanted newline between the subject line and the "Content-Type:" declaration.

So with tremendous courage I embarked on a search of ns_sendmail.tcl - emboldened by Tilman's patches that I applied yesterday! - And guess what I found.......

There is an extra '\n' included before the body which confuses MS Outlook.

So if anyone wants to send html emails from the ACS and doesn't mind hacking ns_sendmail..................
Modified version posted below - includes Tilman's patches.
Mime header and html body need to go into email-password-2.tcl

proc _ns_sendmail {smtp smtpport timeout tolist bcclist \
	from subject body extraheaders} {
    
    ## Put the tolist in the headers
    set rfcto [join $tolist ", "]
    
    ## Build headers
    set msg "To: $rfcto\nFrom: $from\nSubject: $subject\nDate: [ns_httptime [ns_time]]"

    ## Insert extra headers, if any (not for BCC)
    if ![string match "" $extraheaders] {
	set size [ns_set size $extraheaders]
	for {set i 0} {$i < $size} {incr i} {
	    append msg "\n[ns_set key $extraheaders $i]: [ns_set value $extraheaders $i]"
	}
    }

    ## Blank line between headers and body

    append msg "$body\n"

    ## Terminate body with a solitary period
    foreach line [split $msg "\n"] {
	if [string match . $line] {
	    append data .
	}
	append data $line
	append data "\r\n"
    }
    append data .

    ## Open the connection
    set sock [ns_sockopen $smtp $smtpport]
    set rfp [lindex $sock 0]
    set wfp [lindex $sock 1]

    ## Strip "from" email address
    regexp {.*<(.*)>} $from ig from

    ## Perform the SMTP conversation
    if { [catch {
	_ns_smtp_recv $rfp 220 $timeout
	_ns_smtp_send $wfp "HELO [ns_info hostname]" $timeout
	_ns_smtp_recv $rfp 250 $timeout
	_ns_smtp_send $wfp "MAIL FROM:<$from>" $timeout
	_ns_smtp_recv $rfp 250 $timeout

	## Loop through To list via multiple RCPT TO lines
	foreach toto $tolist {
	    _ns_smtp_send $wfp "RCPT TO:<$toto>" $timeout
	    _ns_smtp_recv $rfp 250 $timeout
	}

	## Loop through BCC list via multiple RCPT TO lines
	## A BCC should never, ever appear in the header.  Ever.  Not even.
	foreach bccto $bcclist {
	    _ns_smtp_send $wfp "RCPT TO:<$bccto>" $timeout
	    _ns_smtp_recv $rfp 250 $timeout
	}

	## Loop through To and BCC list via multiple RCPT TO lines
	## A BCC should never, ever appear in the header.  Ever.  Not even.
	foreach toto [concat $tolist $bcclist] {
	    # Transform "Fritz #fritz@foo.com>" into "fritz@foo.com".
	    regexp {.*<(.*)>} $toto ig toto

	    _ns_smtp_send $wfp "RCPT TO:<$toto>" $timeout
	    _ns_smtp_recv $rfp 250 $timeout
	}

	_ns_smtp_send $wfp DATA $timeout
	_ns_smtp_recv $rfp 354 $timeout
	_ns_smtp_send $wfp $data $timeout
	_ns_smtp_recv $rfp 250 $timeout
	_ns_smtp_send $wfp QUIT $timeout
	_ns_smtp_recv $rfp 221 $timeout
    } errMsg ] } {
	## Error, close and report
	close $rfp
	close $wfp
	return -code error $errMsg
    }

    ## Close the connection
    close $rfp
    close $wfp
}

Regards
Richard
Collapse
7: Re: html in emails. (response to 6)
Posted by Richard Hamilton on
The simple modification above broke the basic default emails for confirming registrations to I added this simple check to ns_sendmail and it seems to be working fine. I have only just started working with tcl so my repertoire of instructions is not good. Perhaps the tcl experts could let me know if this is an efficient way to construct it.
if {[string index $body 0] != "\n"} {
    	set newline "\n"
    } else {
    	set newline ""
    }

    append msg $newline "$body\n"

I was not sure if I had to explicitly tell the interpreter that newline contains null so I thought I had better for safety's sake.
Regards
Richard
Collapse
8: Re: html in emails. (response to 1)
Posted by Richard Hamilton on
Please accept my apologies as I gradually ascend the learning curve. Please disregard the previously proposed ugly hack!

This post is aimed at the non expert - experts please smile smugly!

ns_sendmail can receive additional email header declarations which can be passed when the proc is called. The variable that receives the header that you want to pass is called 'extraheaders' in ns_sendmail and the functionality is implemented using an aolserver tcl api call - ns_set.

To specify a single part html email you need to use ns_set in the calling procedure to set a (or multiple) key-value pair(s) to contain the appropriate header(s), and then pass the set to ns_sendmail by adding the name of your new key-value pair set to the list of parameters to be passed. ns_sendmail will then add the header to the email in the appropriate way and everyone will live happily ever after. Example below specifies that the email is html and will satisfy the dreaded MS Outlook.

set extraheaders [ns_set create headers]
ns_set put $extraheaders "Content-Type" "test/html;"


Then call ns_sendmail in the usual way wrapped in a 'catch' :

if [catch {ns_sendmail $email $system_owner $subject $body $extraheaders} errmsg] {
etc.....


Regards
Richard
Collapse
9: Re: html in emails. (response to 8)
Posted by Alfred Werner on
just noticed this -

in the example - it should NOT be

ns_set put $extraheaders "Content-Type" "test/html;"

but rather

ns_set put $extraheaders "Content-Type" "text/html;"

i.e. not TEST, but TEXT

Collapse
10: Re: html in emails. (response to 1)
Posted by Richard Hamilton on
Good spot - furious and inaccurate typing!
Collapse
11: Re: html in emails. (response to 10)
Posted by Alfred Werner on
Well - the reason I noticed is I am implementing this for a client who wants his notifications to go out with his company's look and feel. I have it working, just trying to see  who else has played with this and whether there has been enough interest to post a patch. Being a pine user mainly, I of course want the text/plain to not break :)

I'm thinking of adding a patch to acs-mail-lite to add a parameter with the full path of a single mail template which gets ns_adp_parse'd with just one variable substitution .. <%= $body %> and included as multipart/alternative.

It's acs-mail-LITE so I'm not trying to kill an ogre, just add some simple functionality.

What did you come up with on your end?

Collapse
12: Re: html in emails. (response to 1)
Posted by Richard Hamilton on
Well it is a while ago now but I seem to recall that I modified the qmail calls to call ns_sendmail, created mail templates for the e-mails and then passed the extra header name value pairs using the ns_set functionality built into ns_sendmail.

Beware though that there are a number of mods to ns_sendmail, however I think that I documented my tracks through my daft questions which followed equally daft errors. You should find everything by looking through my posts.

Hope this helps, if I have time I will refresh my memory and post another message.

Regards
Richard

Collapse
13: Re: html in emails. (response to 1)
Posted by Timo Hentschel on
We enhanced acs-mail-lite and wrote a mailing-list package for greenpeace that Malte's gonna upload soon to the HEAD in the contrib-section.

What does it do? The mailing-list package enables you to send plaintext/html/multipart/alternative mails with/without any attachments to a group of users.

We enhanced acs-mail-lite to provide a bounce management support, i.e. it tracks bounced emails and sets users.email_bouncing_p='t' if mails keep bouncing for the same user. It also provides a service-contract that's been called for every bounced mail so that other packages can hook in and execute their custom code for dealing with bouncing mails (the same mechanism could be used to insert email-replies to bboard-postings in the db).

Collapse
14: Re: html in emails. (response to 13)
Posted by Alfred Werner on
That's great news! So I assume that's coming in before the 5.0 feature freeze?
Collapse
15: Re: html in emails. (response to 14)
Posted by Malte Sussdorff on
We face some challenges with General Comments installation in the latest checkout, but once we get around this issue and can test against latest CVS, we will upload. Let's say by wednesday ?
Collapse
16: Re: html in emails. (response to 15)
Posted by Alfred Werner on
Awesome :)
Collapse
18: Re: bounce management (response to 13)
Posted by Doug Harris on
How are you managing bounces? Have you configured OACS to handle SMTP traffic on port 25? Or is your mailserver running a script which interacts with the database?

I've considered something like this, in combination with an extra header, something like "X-OACS-USER-ID: 12345". Since bounced mail usually returns with the full headers, finding this pattern and flagging something in the db is easier than doing the pattern match on email.

Or are you doing something else entirely?

I'm most curious.

Collapse
17: Re: html in emails. (response to 1)
Posted by Janine Ohmer on
Hey everyone, sorry for coming to this thread late.

A while back I modified bulk-mail to handle HTML mail.  This was based on work Mohan did for Sloan, but I made it a bit more generic at Don's request.

You can look at the proc "sweep" in tcl/bulk-mail-procs.tcl, in HEAD, to see how it was done;  it could easily be done the same way in other places.  Most of the work is done by procs in acs-tcl/tcl/mime-procs.tcl.

I had thought of one more thing I could do to make it even easier, but got distracted by some crisis or other and never got back to it.  I will have to see if I can find any notes on what I was thinking of doing.

Collapse
19: Re: html in emails. (response to 1)
Posted by Timo Hentschel on
We added an X-Envelope-From header composed out of "bounce-$user_id-$hashed_message_id-$package_id@$server_url".
We then added a linux user "bounce", configured the postfix seperator "-" and changed the postfix maildir format so that mails will be stored in seperate files (instead of mbox format). This puts all incoming mail beginning with "bounce-*" in a dedicated maildir, i.e. /home/bounce/Maildir. A scheduled proc will then process every newly bounced mail, determine the user_id and the package_id from the bounce adress and call the package-specific service-contract, if one exists.

The same process (scanning for incoming mails) could be used to stuff email-replies to bboard-postings in the appropriate forum...

Collapse
20: Re: html in emails. (response to 19)
Posted by Alfred Werner on
Just as a by-the-way there is a good shell script called decomposemail that will split mbox into separate files.

I've used it lately to split things out to train my spam filters. Thank god for spam filters :) Point being if you can't reconfig to a maildir format, you can just split them out ex post facto.