Forum OpenACS Q&A: ns_sendmail replacement?

Collapse
Posted by Tilmann Singer on
I would like two things that ns_sendmail doesn't offer:

Allow for email address specification in this format: "Fritz Foo <mailto:fritz@foo.com>" alternatively to "mailto:fritz@foo.com";. I don't know which RFC that is in, but I am pretty sure that's some kind of standard.

The first format results in display of the full name in most email programs instead of the email address.

Second, Message-Id generation. Look at the mails you get from openacs.org - old bboard or forums alike - and check the headers. You will see that they do _not_ contain a Message-Id header. Now look at some other mails in your inbox. Most of them will have such a header. I remember reading some RFC where it says while it is not mandatory to include this header it is highly recommended. It will be used for example to build the References: and In-Reply-To: headers when replying to mails. These headers are then essential for mail programs to build a threaded view of the mails (there are some workarounds involving the same subject line, but the way it is intended actually is via these headers).

Why doesn't the MTA create the header after ns_sendmail delivers the message? I am not sure, but I believe that it depends on the MTA, and that at least qmail seems to believe that since ns_sendmail communicates with it via a SMTP socket connection, that it is another MTA and the mail should not be manipulated.

When using the local program qmail-inject to send a mail, qmail does insert a Message-Id header if there is none in the mail yet.

I would like to have (and write if necessary) a ns_sendmail replacement that takes the same arguments as ns_sendmail, optionally accepts a full name included in the address, and generates and returns a Message-Id.

It could propably be made pluggable with different implementations, so that on one installation it uses the local qmail-inject program, sendmail on others, or even ns_sendmail (discarding options that ns_sendmail doesn't support), depending on a parameter setting.

There is already a proc, simply called qmail, in the ecommerce package that uses qmail-inject, which could be used as a base.

Thoughts? Would it be ok to integrate this into acs-mail-lite?

Collapse
Posted by Ben Adida on
Tilmann: it would be a *great* idea to integrate this into acs-mail-lite. That's exactly the idea of acs-mail-lite: to be a complete replacement for ns_sendmail.

If you have any questions about acs-mail-lite, post here, or send email to me and Eric.

Collapse
Posted by Tom Jackson on

It would be best to dump it asap to a real mail program. Using qmail-inject does this. What can be done for other popular choices?

Collapse
Posted by Mat Kovach on
One real problem that might happen is you'll make OACS dependant on qmail.  For some people this might be a
problem.  Some people can react rather badly to running qmail (but are happy running buggy and insecure software, go figure).

Now, there is the mess822 package (http://cr.yp.to/mess822.html) which includes "new-inject" which you can run with the -n option to create a correctly formatted message with message id and output it to standard output, which could be use to feed to any sendmail type wrapper.

We could add the full name by formatting the from header
like:

From: OpenACS Q/A Forum <mailto:notification-56476-2960@openacs.org>

Instead of

From: mailto:notification-56476-2960@openacs.org

Collapse
Posted by Tilmann Singer on
I wouldn't dare making it _require_ qmail, just as an option. It should be possible to build it in a way that it can still call ns_sendmail or dump it to another mailer using the appropriate command instead of qmail-inject.

The forums notifications could really use nicer From: addresses. I thought that maybe instead of

From: OpenACS Q/A Forum <mailto:notification-56476-2960@openacs.org>

they could even be set to the posters full name, as it is the case with mails from most mailing lists. Then it'd just need a Reply-To: mailto:notification-1234@openacs.org header so that replies go to the notifications incoming email handler if this is set up.

Collapse
Posted by Roberto Mello on
I'd concur that requiring qmail is not a good move.

I also agree with the nicer from lines that til suggested. I'll see about hacking notifications to take on a more mailing-list-like behavior (From: email-of-poster <name-of-poster>, take some of the lines off the beginning of message, appending forum/message/subscription info after a "--" at the end of the message) when you don't want the reply-via-email feature to work. Currently there seems to be no way to disable that.

Collapse
Posted by Jonathan Ellis on
actually, I prefer having From: mailto:notifications_robot@openacs.org or what have you, and reply_to set to the author.  Makes it much easier to filter.
Collapse
Posted by Ben Adida on
qmail should definitely not be required - even though mbox format is currently required for parsing incoming emails (that's my fault - I couldn't figure out a way to get this done quickly without that).

We should probably have some kind of service contract to allow a qmail provider, a sendmail provider, etc...

Collapse
Posted by Tilmann Singer on
Roberto: great! I have put a few bugs in sdm: http://sdm.openacs.org/sdm/open-bafs.tcl?module_id=129&package_id=9 - hopefully you'll be able to close them all ;)

Jonathan: While it might be easier to filter, I think it's much nicer to see who the posting is actually from instead of some more or less meaningless robot address. Consider the way most mailing lists work - they have the poster in From: and sometimes add a little list indicator in square brackets in front of the subject, like this: "Re: [foo-devel] the actual subject". Maybe this should be optionally added to forums, don't know. But even without the square bracket indicator I'd rather have the posters originator in the From column of my mail client.

Ben: I didn't suggest to require qmail, rather a wrapper proc like acs_mail_lite::sendmail that calls an implementation like acs_mail_lite::sendmail_impl_qmail or acs_mail_lite::sendmail_impl_ns_sendmail, depending on a parameter setting. Code example follows.

Also I think you meant Maildir format, not mbox for incoming email, afaik that's different.

Collapse
Posted by Ben Adida on
Tilmann: absolutely, you're right, I meant maildir. It's one of those days :)

Yes, I think you have the right concept with different implementations, but it would be good to implement using service-contracts that standardize that type of API/provider interface. We can chat about it on IRC, if you'd like.

Collapse
Posted by Don Baccus on
SERVICE CONTRACTS.  Yes, yes, and yes again.

Jeff Davis is picking up that package, BTW, and we've chatted about having it enforce service contracts and also about providing a high-level means to define contracts and their implementations (i.e. remove the redudantly boring coding task of writing definitions of methods and their parameters as a set of INSERT SQL statements in both PG and Oracle).

Anyway ... ns_sendmail is written in Tcl, Til, and if you look you'll see it depends on some low-level SMTP protocol handling Tcl procs.

Beyond the issues you raise ns_sendmail is slow as it always builds a socket, does the SMTP stuff, then tears down the socket for the given message.  This is fine for single messages (even those with lots of recipients, of course) but not for a bunch of messages sent out at once, say a bunch of daily digests built for a bunch of forums.

It would be quite simple to bulk-mail by plugging into the low-level SMTP code and only opening the socket once while sending everything that needs sending.

Collapse
Posted by MaineBob OConnor on

    Don wrote: Beyond the issues you raise ns_sendmail is slow as it always builds a socket...

So this is why when I loop thru 15,000 emails to using ns_sendmail that we use so much cpu time... I've taken to using a 1 second delay so that web service performance doesn't suffer.

We use Postfix and really like it better than Qmail. I understand that postfix can be configured to use Qmail type mailboxes. I don't know about the postfix equivelent of qmail-inject.

    Don: It would be quite simple to bulk-mail by plugging into the low-level SMTP code and only opening the socket once while sending everything that needs sending.

Great and I'd like to see this and for now would like to back port it to openacs 3.x until I see a clear path to migrate my current sites to 4.x.

-Bob

Collapse
Posted by Tilmann Singer on
I don't see a problem writing a service contract to send a single mail message. It would consist of a single function that accepts more or less the same arguments as ns_sendmail with a few additional ones, such as from_pretty_name etc.

This would imply creating a package for each implementation, like acs-mail-lite-qmail, acs-mail-lite-ns_sendmail etc., and configuring the implementation to use via the service contract binding UI, right?

I don't see an obvious solution for a service contract that does something like sending several different messages in a row, though, which would be necessary to enable an implementation that e.g. keeps an SMTP socket open in the background. Would it require to add something like a 'handle' that the caller has to open and specify when sending a mail? Or would it be possible to build a function that takes several mails at once as arguments - also as many as 15000? Would this require some upvar trickery, such as in the search service contracts?

What are the arguments against integrating separate transport implementations in acs-mail-lite directly? I guess there won't be more than 5 of them realistically. Initially maybe ns_sendmail, postfix and qmail - would that really be that unmaintainable?

Collapse
Posted by Bart Teeuwisse on

A related problem is that ns_sendmail (in aolserver/modules/tcl/sendmail.tcl) discards everything but the <notification-56476-2960@openacs.org> part of an e-mail address.

The notifications package is configured to send out messages from [address_domain] mailer <[reply_address_prefix]@[address_domain]. In the case of OpenACS these messages would show up as coming from 'OpenACS mailer' if it wasn't for ns_sendmail.

My vote goes to 'fixing' ns_sendmail, adding a Reply-To header to notifications and replacing the from address with the name and e-mail of the poster (for forums).

/Bart

Collapse
Posted by Talli Somekh on
Bart said: "My vote goes to 'fixing' ns_sendmail, adding a Reply-To header to notifications and replacing the from address with the name and e-mail of the poster (for forums)."

Bart, you said you have a fix for this? I would like to use it, but actually not for forums but for notifications-enabled news.

Do you have a patch or something?

TIA

talli

Collapse
Posted by Bart Teeuwisse on

Talli,

Replace ns_sendmail in aolserver/modules/tcl/sendmail.tcl with:


proc ns_sendmail { to from subject body {extraheaders {}} {bcc {}} } {

    ## Takes comma-separated values in the "to" parm
    ## Multiple To and BCC addresses are handled appropriately.
    ## Original ns_sendmail functionality is preserved.

    ## Cut out carriage returns
    regsub -all "\n" $to "" to
    regsub -all "\r" $to "" to
    regsub -all "\n" $bcc "" bcc
    regsub -all "\r" $bcc "" bcc
    
    ## Split to into a proper list
    set tolist_in [split $to ","]
    set bcclist_in [split $bcc ","]
    
    ## Get smtp server into, if none then use localhost
    set smtp [ns_config ns/parameters smtphost]
    if [string match "" $smtp] {
	set smtp [ns_config ns/parameters mailhost]
    }
    if [string match "" $smtp] {
	set smtp localhost
    }
    set timeout [ns_config ns/parameters smtptimeout]
    if [string match "" $timeout] {
	set timeout 60
    }
    set smtpport [ns_config ns/parameters smtpport]
    if [string match "" $smtpport] {
	set smtpport 25
    }

    ## Extract "from" email address
#     if [regexp {.*<(.*)>} $from ig address] {
# 	set from $address
#     }
    
    set tolist [list]
    foreach toaddr $tolist_in {
# 	if [regexp {.*<(.*)>} $toaddr ig address] {
# 	    set toaddr $address
# 	}
	lappend tolist "[string trim $toaddr]"
    }
    
    set bcclist [list]
    if ![string match "" $bcclist_in] {
	foreach bccaddr $bcclist_in {
# 	    if [regexp {.*<(.*)>} $bccaddr ig address] {
# 		set bccaddr $address
# 	    }
	    lappend bcclist "[string trim $bccaddr]"
	}
    }
    
    ## Send it along to _ns_sendmail
    _ns_sendmail $smtp $smtpport $timeout $tolist $bcclist \
	    $from $subject $body $extraheaders
}

The fix is in the 3 groups of 3 lines commented out.

/Bart

Collapse
Posted by Talli Somekh on
Thanks Bart!

talli

Collapse
Posted by Tilmann Singer on
I didn't try the code out but from looking at it I wonder if it does the right thing: by commenting out these lines you get  a list like this in tolist: {{Fritz <mailto:fritz@frotz.com>} mailto:foo@bar.com}.

Later in _ns_sendmail there is this code:

## Loop through To list via multiple RCPT TO lines
foreach toto $tolist {
    _ns_smtp_send $wfp "RCPT TO:<$toto>" $timeout
    _ns_smtp_recv $rfp 250 $timeout
}

which means it would for the first item in the $tolist send the following line to the SMTP server:

RCPT TO:<Fritz <mailto:fritz@frotz.com>>

which has one enclosing pair of angle brackets too much. Shouldn't that be:

RCPT TO: Fritz <mailto:fritz@frotz.com>

? You might get away with it if it is a forgiving mailserver, but it isn't right IMHO. Or am I missing something?

Collapse
Posted by Jeff Davis on
I looked at rfc 2821 and it looks like Til is right. _ns_sendmail should do the stripping ns_sendmail was doing before sending the "MAIL FROM:" and "RCPT TO:" messages to the SMTP server. The "To:" and "From:" headers in the message body on the other hand should have the fill mail addresses in them.

Here is a somewhat relevant example from the RFC (it has explicit relaying for the MAIL FROM though):

      S: 220 xyz.com Simple Mail Transfer Service Ready
      C: EHLO foo.com
      S: 250 xyz.com is on the air
      C: MAIL FROM:<@foo.com:JQP@bar.com>
      S: 250 OK
      C: RCPT TO:<ones@XYZ.COM>
      S: 250 OK
      C: DATA
      S: 354 Start mail input; end with <CRLF>.<CRLF>
      C: Received: from bar.com by foo.com ; Thu, 21 May 1998
      C:     05:33:29 -0700
      C: Date: Thu, 21 May 1998 05:33:22 -0700
      C: From: John Q. Public <JQP@bar.com>
      C: Subject:  The Next Meeting of the Board
      C: To: Jones@xyz.com
      C:
      C: Bill:
      C: The next meeting of the board of directors will be
      C: on Tuesday.
      C:                         John.
      C: .
      S: 250 OK
      C: QUIT
      S: 221 foo.com Service closing transmission channel

Collapse
Posted by Bart Teeuwisse on

Til is right indeed.

It is a suprise that my qmail mailer has been so forgiving.

To remain RFC 2821 compliant, the extraction of the e-mail addresses should be moved from ns_sendmail to _ns_sendmail like so:


proc ns_sendmail { to from subject body {extraheaders {}} {bcc {}} } {

    ## Takes comma-separated values in the "to" parm
    ## Multiple To and BCC addresses are handled appropriately.
    ## Original ns_sendmail functionality is preserved.

    ## Cut out carriage returns
    regsub -all "\n" $to "" to
    regsub -all "\r" $to "" to
    regsub -all "\n" $bcc "" bcc
    regsub -all "\r" $bcc "" bcc

    ## Split to into a proper list
    set tolist_in [split $to ","]
    set bcclist_in [split $bcc ","]

    ## Get smtp server into, if none then use localhost
    set smtp [ns_config ns/parameters smtphost]
    if [string match "" $smtp] {
        set smtp [ns_config ns/parameters mailhost]
    }
    if [string match "" $smtp] {
        set smtp localhost
    }
    set timeout [ns_config ns/parameters smtptimeout]
    if [string match "" $timeout] {
        set timeout 60
    }
    set smtpport [ns_config ns/parameters smtpport]
    if [string match "" $smtpport] {
        set smtpport 25
    }

    set tolist [list]
    foreach toaddr $tolist_in {
        lappend tolist "[string trim $toaddr]"
    }

    set bcclist [list]
    if ![string match "" $bcclist_in] {
        foreach bccaddr $bcclist_in {
            lappend bcclist "[string trim $bccaddr]"
        }
    }

    ## Send it along to _ns_sendmail
    _ns_sendmail $smtp $smtpport $timeout $tolist $bcclist \
            $from $subject $body $extraheaders
}

proc _ns_sendmail {smtp smtpport timeout tolist bcclist \
        from subject body extraheaders} {

    ## Put the tolist in the headers
    set rfcto [join $tolist ", "]

    ## Build headers
    set msg "To: $rfcto\nFrom: $from\nSubject: $subject\nDate: [ns_httptime [ns_time]]"

    ## Insert extra headers, if any (not for BCC)
    if ![string match "" $extraheaders] {
        set size [ns_set size $extraheaders]
        for {set i 0} {$i < $size} {incr i} {
            append msg "\n[ns_set key $extraheaders $i]: [ns_set value $extraheaders $i]"
        }
    }

    ## Blank line between headers and body
    append msg "\n\n$body\n"

    ## Terminate body with a solitary period
    foreach line [split $msg "\n"] {
        if [string match . $line] {
            append data .
        }
        append data $line
        append data "\r\n"
    }
    append data .

    ## Open the connection
    set sock [ns_sockopen $smtp $smtpport]
    set rfp [lindex $sock 0]
    set wfp [lindex $sock 1]

    ## Perform the SMTP conversation
    if { [catch {
        _ns_smtp_recv $rfp 220 $timeout
        _ns_smtp_send $wfp "HELO AOLserver [ns_info hostname]" $timeout
        _ns_smtp_recv $rfp 250 $timeout

        ## Extract "from" email address
        if [regexp {.*<(.*)>} $from ig address] {
            set from $address
        }
        _ns_smtp_send $wfp "MAIL FROM: $from" $timeout
        _ns_smtp_recv $rfp 250 $timeout

        ## Loop through To list via multiple RCPT TO lines
        foreach toto $tolist {
            if [regexp {.*<(.*)>} $toto ig address] {
                set toto $address
            }
            _ns_smtp_send $wfp "RCPT TO: $toto" $timeout
            _ns_smtp_recv $rfp 250 $timeout
        }

        ## Loop through BCC list via multiple RCPT TO lines
        ## A BCC should never, ever appear in the header.  Ever.  Not even.
        foreach bccto $bcclist {
            if [regexp {.*<(.*)>} $bccto ig address] {
                set bccto $address
            }
            _ns_smtp_send $wfp "RCPT TO: $bccto" $timeout
            _ns_smtp_recv $rfp 250 $timeout
        }

        _ns_smtp_send $wfp DATA $timeout
        _ns_smtp_recv $rfp 354 $timeout
        _ns_smtp_send $wfp $data $timeout
        _ns_smtp_recv $rfp 250 $timeout
        _ns_smtp_send $wfp QUIT $timeout
        _ns_smtp_recv $rfp 221 $timeout
    } errMsg ] } {
        ## Error, close and report
        close $rfp
        close $wfp
        return -code error $errMsg
    }

    ## Close the connection
    close $rfp
    close $wfp
}

/Bart

Collapse
Posted by Tilmann Singer on
I don't think that's right:
        ## Loop through To list via multiple RCPT TO lines
        foreach toto $tolist {
            if [regexp {.*<(.*)>} $toto ig address] {
                set toto $address
            }
            _ns_smtp_send $wfp "RCPT TO: $toto" $timeout
            _ns_smtp_recv $rfp 250 $timeout
        }
The rfc specifies it like this:
MAIL FROM:<userx@y.foo.org>
RCPT TO:<userc@d.bar.org>
so aolserver would have to do that:
        ## Loop through To list via multiple RCPT TO lines
        foreach toto $tolist {
            if [regexp {.*<(.*)>} $toto ig address] {
                set toto $address
            }
            _ns_smtp_send $wfp "RCPT TO:<$toto>" $timeout
            _ns_smtp_recv $rfp 250 $timeout
        }
and while we're at it we might as well join the processing of the $tolist above and the $bcclist - they do the same.
Collapse
Posted by Tilmann Singer on
Here's a patch against the original sendmail.tcl that hopefully includes all corrections. I'll try to submit it on aolserver's bug tracker on sourceforge (last time I tried it was not working due to timeouts).

*** root/aolserver/tcl/sendmail.tcl	Fri Aug 11 22:04:10 2000
--- aolserver/modules/tcl/sendmail.tcl	Fri Jan 17 12:34:59 2003
***************
*** 96,120 ****
  	set smtpport 25
      }
  
-     ## Extract "from" email address
-     if [regexp {.*<(.*)>} $from ig address] {
- 	set from $address
-     }
-     
      set tolist [list]
      foreach toaddr $tolist_in {
- 	if [regexp {.*<(.*)>} $toaddr ig address] {
- 	    set toaddr $address
- 	}
  	lappend tolist "[string trim $toaddr]"
      }
      
      set bcclist [list]
      if ![string match "" $bcclist_in] {
  	foreach bccaddr $bcclist_in {
- 	    if [regexp {.*<(.*)>} $bccaddr ig address] {
- 		set bccaddr $address
- 	    }
  	    lappend bcclist "[string trim $bccaddr]"
  	}
      }
--- 96,109 ----
***************
*** 160,165 ****
--- 149,157 ----
      set rfp [lindex $sock 0]
      set wfp [lindex $sock 1]
  
+     ## Strip "from" email address
+     regexp {.*<(.*)>} $from ig from
+ 
      ## Perform the SMTP conversation
      if { [catch {
          _ns_smtp_recv $rfp 220 $timeout
***************
*** 168,183 ****
  	_ns_smtp_send $wfp "MAIL FROM:<$from>" $timeout
  	_ns_smtp_recv $rfp 250 $timeout
  	
! 	## Loop through To list via multiple RCPT TO lines
! 	foreach toto $tolist {
! 	    _ns_smtp_send $wfp "RCPT TO:<$toto>" $timeout
! 	    _ns_smtp_recv $rfp 250 $timeout	
! 	}
! 	
! 	## Loop through BCC list via multiple RCPT TO lines
  	## A BCC should never, ever appear in the header.  Ever.  Not even.
! 	foreach bccto $bcclist {
! 	    _ns_smtp_send $wfp "RCPT TO:<$bccto>" $timeout
  	    _ns_smtp_recv $rfp 250 $timeout
  	}
  	
--- 160,173 ----
          _ns_smtp_send $wfp "MAIL FROM:<$from>" $timeout
          _ns_smtp_recv $rfp 250 $timeout
          
!         ## Loop through To and Bcc list via multiple RCPT TO lines
          ## A BCC should never, ever appear in the header.  Ever.  Not even.
!         foreach toto [concat $tolist $bcclist] {
! 
!             # Transform "Fritz <fritz@foo.com>" into "fritz@foo.com".
!             regexp {.*<(.*)>} $toto ig toto
! 
!             _ns_smtp_send $wfp "RCPT TO:<$toto>" $timeout
              _ns_smtp_recv $rfp 250 $timeout    
          }
          
Collapse
Posted by Lars Pind on
I don't like the fact that each OACS installation needs to hand-patch AOLserver, no matter how easy it is to do.

Is there a chance that we could either get it in to the official AOLserver distributions -- 3.3, 3.5, and 4.0 -- or create our own OACS version such as ad_sendmail, and make sure to use that everywhere instead.

/Lars

Collapse
Posted by Jeff Davis on
We could just
rename ns_sendmail ns_sendmail_questionable
rename ns_ourversion ns_sendmail
(which we probably should have done with things like ad_returnredirect and such).

That way if (when) they fix it, we can just take it out.

Collapse
Posted by Tilmann Singer on
Ok, I reported a bug on aolserver 3.3 and uploaded the patch. Shortly after that I discovered that Bartt reported the same bug one day ago, oops.

For the meantime I agree it would make sense to redefine ns_sendmail with our own version. Should I add this to acs-tcl? And in order to not annoy OpenACS users once it is fixed I would make it a parameter setting (acs-tcl -> RedefineNsSendmailP) default 0 - any better suggestions?

Collapse
Posted by Richard Hamilton on
I have been having problems with aolserver being unable to send a password reminder email to the site admin. I am running sendmail on localhost.

I have used telnet to connect and have carried out the MAIL FROM: and RCPT TO: tests, all of which work fine. If I use emacs to send a test email as nsadmin it works just fine.

However, if I ask the server to send me a password reminder I get an error :

Now we're really in trouble because we got this error: 
    Expected a 250 status line; got:
501 5.0.0 Invalid domain name
  
when trying to send you the following email: 
Subject: Your forgotten password on The Militaria Exchange

Please follow the following link to reset your password:

https://63.246.3.10:8446/user/password-update?password%5fold=652294FA3&user%5fid=2604

It seems that aolserver is talking to sendmail ok and sendmail is sending back the Invalid domain name message.

Is this the problem that Tillman's patch fixes or is my setup at fault?

Regards
Richard
Collapse
Posted by Richard Hamilton on
Have applied Tilmann's hand patch but still getting this error :
Now we're really in trouble because we got this error: 
    Expected a 250 status line; got:
501 5.0.0 Invalid domain name
  
when trying to send you the following email: 
Subject: Your forgotten password on The Militaria Exchange

Please follow the following link to reset your password:

https://63.246.3.10:8446/user/password-update?password%5fold=DF9F0A86B&user%5fid=2604
Any ideas?

Regards
Richard
Collapse
Posted by Jeff Davis on
That error is what you get when your EHLO or HELO host name is messed up. There was a bug (fixed now in 4.0) where it would send
EHLO AOLServer yourserver.com
rather than simply "EHLO yourserver.com". The fix is in aolserver/tcl/ns_sendmail.tcl and you can see it in CVS at aolserver.

Another way this can be broken is if you send the hostname with a port. e.g.

$ telnet 127.0.0.1 25
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
220 test.xorch.net ESMTP Sendmail 8.11.6/8.11.6; Thu, 6 Feb 2003 16:58:11 GMT
EHLO xarg.net:8080
501 5.0.0 Invalid domain name
I think it sends the hostname from the config.tcl file which should not have the port on it iirc.
Collapse
Posted by Richard Hamilton on
Great thanks - I'll try it. I noticed the lines that sent AOLserver but Tilmann's patch seemed to imply that these should be left as is.
Regards
Richard
Collapse
Posted by Richard Hamilton on
FANTASTIC - THANK YOU!!!
Regards
Richard
Collapse
Posted by Lars Pind on
Jeff said:

That error is what you get when your EHLO or HELO host name is messed up. There was a bug (fixed now in 4.0) where it would send
EHLO AOLServer yourserver.com

rather than simply "EHLO yourserver.com". The fix is in aolserver/tcl/ns_sendmail.tcl and you can see it in CVS at aolserver.

In case you're using sendmail, and you'd rather change your sendmail configuration than mock with AOLserver, here's the line that you'll need to add to your sendmail.mc file on Red Hat Linux 8.0:

define(`confALLOW_BOGUS_HELO',`True')dnl

/Lars

Collapse
Posted by Andrew Piskorski on
This is quite the long thread, but I think no one has yet added generation of the Message-Id in any of the versions of ns_sendmail above? Hm, at first glance, looks like the AOLserver 4.0 beta sendmail.tcl has some slight improvements over 3.3, but still doesn't do that. Well, I hacked my ns_sendmail to do some simply Message-Id generation a year or two ago to do that. It wasn't hard, but FYI, here's the code:

My _ns_smtp_recv, _ns_smtp_send, and ns_sendmail procs are unchanged from the versions shipped with 3.3+ad13. I just added this extra top level nsv command, and changed _ns_sendmail a bit. Note there are only a few lines of changed code here! (But I find trying to read diffs in BBoard annoying, so I didn't do that.) After the

if { ! $message_id_already_done_p }
line it's all stock, except for some comments:

if { ![nsv_exists ns_sendmail sequence] } {
   nsv_set ns_sendmail sequence 0
}

proc _ns_sendmail {smtp smtpport timeout tolist bcclist \
        from subject body extraheaders} {
    
    ## Put the tolist in the headers
    set rfcto [join $tolist ", "]
    
    ## Build headers
    set msg "To: $rfcto\nFrom: $from\nSubject: $subject\nDate: [ns_httptime [ns_time]]"
    
    ## Insert extra headers, if any (not for BCC)
    set message_id_already_done_p 0
    if ![string match "" $extraheaders] {
        set size [ns_set size $extraheaders]
        for {set i 0} {$i < $size} {incr i} {
           set key [ns_set key $extraheaders $i]
           # This is rather hack-ish...
           if { [string compare $key {Message-ID}] == 0 } {
              set message_id_already_done_p 1
           }
           append msg "\n${key}: [ns_set value $extraheaders $i]"
        }
    }

    # Insert a unique "Message-ID:" header, but only if the caller did
    # not manually include a Message-ID header:
    #
    # An application could use the Message-ID header for
    # e.g. threading support, but we're not trying to do anything
    # fancy like that here.  We just want to include a globally-unique
    # ID.  Why?  Well, for one thing, since most email user agents
    # include a Message-ID, but most SPAM software does not, some
    # anti-SPAM software filters out email which does not have a
    # Message-ID...
    #
    # Note: The $message_id below is guaranteed to be globally unique
    # if and only if *ALL* of the following conditions are true:
    #
    # 1. Your unix box's hostname (which is what [ns_info hostname]
    #    returns) is set to a fully-qualified name like
    #    "philip.greenspun.com", NOT just a local hostname like
    #    "philip".
    # 2. Your fully-qualified hostname is in fact globally-unique.
    #    AKA, you didn't do something foolish like set up two separate
    #    machines that both think their hostname is
    #    "philip.greenspun.com".
    # 3. On your unix host, you have only ONE AOLserver running with
    #    the server name returned by [ns_info server].
    # 4. Since [ns_info boottime] is in seconds, you never restart
    #    your AOLserver multiple times in < 1 second, jump your system
    #    clock backwards in time, or etc.
    # 5. Once the "ns_sendmail sequence" nsv variable is set, you
    #    never manually fool with it to re-set it to a previous value.
    #    While the server is running, this value must always increase,
    #    never decrease.
    #
    # --atp@piskorski.com, 2001/10/11 11:51 EDT

    if { ! $message_id_already_done_p } {
       set message_id "[nsv_incr ns_sendmail sequence].[ns_info boottime].[ns_info server]@[ns_info hostname]"
       append msg "\nMessage-ID: $message_id"
    }
    
    ## Blank line between headers and body
    append msg "\n\n$body\n"
    
    ## Terminate body with a solitary period
    foreach line [split $msg "\n"] { 
        if [string match . $line] {
            append data .
        }
        append data $line
        append data "\r\n"
    }
    append data .
    
    ## Open the connection
    set sock [ns_sockopen $smtp $smtpport]
    set rfp [lindex $sock 0]
    set wfp [lindex $sock 1]

    ## Perform the SMTP conversation
    if { [catch {
        _ns_smtp_recv $rfp 220 $timeout
        _ns_smtp_send $wfp "HELO AOLserver [ns_info hostname]" $timeout
        _ns_smtp_recv $rfp 250 $timeout
        _ns_smtp_send $wfp "MAIL FROM:<$from>" $timeout
        _ns_smtp_recv $rfp 250 $timeout

        # TODO: Above, should optionally take a "Return-Path" to use
        # as the envelope sender address (aka, envelope return path)
        # rather than always using $from.  This would allow using
        # VERPs, for instance, as discussed at:
        #   "http://cr.yp.to/proto/verp.txt"
        # See also discussion at:
        #   "http://www.arsdigita.com/bboard/q-and-a-fetch-msg?msg%5fid=000awU"
        # --atp@piskorski.com, 2001/10/11 10:25 EDT
        
        ## Loop through To list via multiple RCPT TO lines
        foreach toto $tolist {
            _ns_smtp_send $wfp "RCPT TO:<$toto>" $timeout
            _ns_smtp_recv $rfp 250 $timeout     
        }
        
        ## Loop through BCC list via multiple RCPT TO lines
        ## A BCC should never, ever appear in the header.  Ever.  Not even.
        foreach bccto $bcclist {
            _ns_smtp_send $wfp "RCPT TO:<$bccto>" $timeout
            _ns_smtp_recv $rfp 250 $timeout
        }
        
        _ns_smtp_send $wfp DATA $timeout
        _ns_smtp_recv $rfp 354 $timeout
        _ns_smtp_send $wfp $data $timeout
        _ns_smtp_recv $rfp 250 $timeout
        _ns_smtp_send $wfp QUIT $timeout
        _ns_smtp_recv $rfp 221 $timeout
    } errMsg ] } {
        ## Error, close and report
        close $rfp
        close $wfp
        return -code error $errMsg
    }

    ## Close the connection
    close $rfp
    close $wfp
}
Collapse
Posted by Andrew Piskorski on
Of course, it might be better to generate a guaranteed globally unique
Message-Id using some sort of hash.  But I didn't spend any time
thinking about that.  Probably best to look at the source of some
other programs generating Message-Id's and see how they do it.
Collapse
Posted by Tilmann Singer on
I've added this to my version of acs-mail-lite ( http://tils.net/acs-mail-lite.tgz ). I hope to be able to integrate this into the toolkit some time.

The advantage of having it in acs-mail-lite is that one can make use of a db sequence, and you can make acs_mail_lite::send return the generated id (or the re-used one if one was specified explicitely), so that the caller can save it for future references, e.g. for generating a References: header.

    ad_proc -private generate_message_id {
        {-sequence_value ""}
    } {
        Generate an id suitable as a Message-Id: header for an email.

        @param sequence_value the value of acs_mail_lite_id_seq to use - when empty a new value will be fetched
    } {

        if { [empty_string_p $sequence_value] } {
            set sequence_value [db_string get_next_message_id "select nextval('acs_mail_lite_id_seq')"]
        }

        # The combination of time accurate to the second and sequence
        # value should be pretty unique. It's unlikely that the
        # sequence gets recreated and two mails are being sent all
        # within one second.
        return "[ns_fmttime [ns_time] "%Y%m%d%H%M%S"].$sequence_value.oacs@[ad_host]"

    }