What kind of 'net connection do you have? You could be clogging your pipe causing the toilet to overflow :)
It's definitely a problem with your MTA - after all, for several minutes afterwards it is refusing connections entirely.
I don't know whether exim's faster or slower than qmail etc etc. However I do know that a properly configured mail server can keep up with a lot of ns_sendmail traffic. Openacs.org, for instance, spams more than 30 users every time one of us posts to the bboard ...
