That error is what you get when your EHLO or HELO
host name is messed up. There was a bug (fixed now in 4.0)
where it would send
EHLO AOLServer yourserver.com
rather than simply "EHLO yourserver.com". The fix is in aolserver/tcl/ns_sendmail.tcl and you can see it in CVS at aolserver.
Another way this can be broken is if you send the hostname
with a port.
e.g.
$ telnet 127.0.0.1 25
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
220 test.xorch.net ESMTP Sendmail 8.11.6/8.11.6; Thu, 6 Feb 2003 16:58:11 GMT
EHLO xarg.net:8080
501 5.0.0 Invalid domain name
I think it sends the hostname from the config.tcl
file which should not have the port on it iirc.