Date: Sun, 14 Mar 1999 20:49:30 -0600 From: Chris Adams To: BUGTRAQ@netspace.org Subject: Microsoft's SMTP service broken/stupid Our mail servers came to a screeching halt today thanks to Microsoft. Our servers are still running sendmail 8.8 (we've got custom stuff and are working on upgrading to 8.9, but it has been slow), so any kind of DNS error (like invalid reverse DNS) returns a 4xx error - temporary problem. When we get a message like this, the sending site will requeue the message and try again in 30 minutes to an hour. After a bit, they stop trying. It is not a perfect solution, but it is all that is available under sendmail 8.8 (sendmail 8.9 differentiates between temporary and permanent DNS errors). Well, that has been fine, but now Microsoft's SMTP service comes along. When it gets that temporary error (for invalid reverse DNS), it tries again. Fast. Like, right away, with no delay. This bogs down our servers a bit, especially the extra logging load, but eventually they go away. Yesterday, we got hit by four different servers running Microsoft's software. One attempted delivery nearly 200,000 times, and the other three attempted to 30,000-40,000 times each. This on a server that usually sees ~40,000 messages a day. This filled up our logs, bogged everything down, and basically killed us. This is not a configuration issue AFAIK. In the past, I've worked through it with one person, and he said he bumped up his retry time to 3 hours and his server was still attempting multiple deliveries per second. This basically amounts to a denial of service attack by Microsoft's SMTP service. Here is the connect string from several of the servers that hit us (I've changed the hostname): 220-example.com Microsoft SMTP MAIL ready at Sun, 14 Mar 1999 21:44:02 -0500 Version: 5.5.1877.977.9 I was able to connect to several of the SMTP servers that hit us and they are all running this version. Several of them don't accept incoming connections (gee thanks - send me junk and don't accept any back). I haven't been able to find anything at Microsoft about this. I would think that attempting several outgoing connections per second would tend to bog down the NT server as well, so I figured they might have mentioned it. Has anybody else seen this? -- Chris Adams - cadams@ro.com System Administrator - Renaissance Internet Services I don't speak for anybody but myself - that's enough trouble. ------------------------------------------------------------------------------- Date: Wed, 17 Mar 1999 14:58:43 -0800 From: "David Lemson (Exchange)" To: BUGTRAQ@netspace.org Subject: Re: Microsoft's SMTP service broken/stupid We have confirmed this as a problem with the Microsoft SMTP Service, as shipped in Windows NT 4 Service Pack 4. We are working on a fix and will have it tested shortly. We will have a Knowledge Base article created as soon as the fix is ready so that people can find the fix when they run into the problem. Any Microsoft customer who is hitting this problem (such as the people whose servers are connecting to you over and over) may e-mail me directly to get set up with the right people to get the fix. The service is erroneously not treating the 4xx error as a reason to defer the delivery until the next queue run. Instead, it treats it as a very transient error and retries immediately. In your case, there is another (easier) solution for the servers that are connecting to you over and over: it sounds like if they were to fix their inverse DNS entries, so you didn't give them a temporary error code, the mail would succeed. This is not to say that what the SMTP Service is doing is right, but there may be another way to solve this particular problem. Another solution, which you allude to, is for your server to issue a permanent (5xx) code to a problem that will not get corrected on its own (such as an invalid inverse DNS record). David Lemson Microsoft SMTP Service Program Manager dlemson@microsoft.com ------------------------------------------------------------------------------- Date: Wed, 17 Mar 1999 19:47:52 +1300 From: Alan Brown To: BUGTRAQ@netspace.org Subject: Re: Microsoft's SMTP service broken/stupid On Sun, 14 Mar 1999, Chris Adams wrote: > Well, that has been fine, but now Microsoft's SMTP service comes along. > When it gets that temporary error (for invalid reverse DNS), it tries > again. Fast. Like, right away, with no delay. This bogs down our > servers a bit, especially the extra logging load, but eventually they go > away. > > Yesterday, we got hit by four different servers running Microsoft's > software. One attempted delivery nearly 200,000 times, and the other > three attempted to 30,000-40,000 times each. This on a server that > usually sees ~40,000 messages a day. This filled up our logs, bogged > everything down, and basically killed us. What's needed is judicious tuning of these configuration items: # load average at which we just queue messages O QueueLA=8 # load average at which we refuse connections O RefuseLA=12 The above two default to 8 and 12 # maximum number of children we allow at one time O MaxDaemonChildren=12 # maximum number of new connections per second O ConnectionRateThrottle=3 If the above two aren't defined, there are no limits. Also useful for load control are these two items: # deliver each queued job in a separate process? #O ForkEachJob # single thread deliveries (requires HostStatusDirectory)? #O SingleThreadDelivery AB ------------------------------------------------------------------------------- Date: Wed, 17 Mar 1999 17:49:57 -0700 From: Bob Beck To: BUGTRAQ@netspace.org Subject: Re: Microsoft's SMTP service broken/stupid "David Lemson (Exchange)" writes: > connecting to you over and over: it sounds like if they were to fix their > inverse DNS entries, so you didn't give them a temporary error code, the > mail would succeed. This is not to say that what the SMTP Service is doing > is right, but there may be another way to solve this particular problem. > > Another solution, which you allude to, is for your server to issue a > permanent (5xx) code to a problem that will not get corrected on its own > (such as an invalid inverse DNS record). You miss the point - this could simply be that their DNS is down or unreachable in a timely manner so the lookup fails. When this produces an nonexistent entry it actually *does* often correct itself "on it's own" once the DNS server is reachable again. Similarly the invalid entry may in fact be corrected before the usual timeout and allow the mail to proceed. That's the whole point of returning 4XX. The only way your "solution" is a solution is to return 5XX errors for *ALL* situations that currently return a 4XX, otherwise we risk being DOS'ed by a poorly written server that doesn't treat SMTP errors right. And this isn't a solution - You lose mail that you shouldn't. So now I should lose mail on a transient DNS failure because microsoft distrbutes code that doesn't play SMTP nicely? I don't think so. The only solution is to fix the buggy code. and/or block access >from sites running buggy code. -Bob ------------------------------------------------------------------------------- Date: Wed, 17 Mar 1999 17:39:16 -0800 From: "David Lemson (Exchange)" To: BUGTRAQ@netspace.org Subject: Re: Microsoft's SMTP service broken/stupid I have gotten a few queries and I realized that I was not completely clear. This bug does not, to our knowledge, affect any version of Microsoft Exchange's Internet Mail Service. It is solely with the SMTP Service component of Internet Information Service (IIS) v4.0.