Greylisting Advantages and Disadvantages

greylistingIn the previous two articles, we covered installation and configuration of spamd, a useful spam-referral daemon. In this article, we will examine some of the advantages and disadvantages of greylisting.

The Greylisting Process

Before we begin, it might be useful to review some of the basic concepts of spam deferral. The process involves, at its most basic level, dividing hosts into three categories: blacklisted hosts, whitelisted hosts, and greylisted hosts. Blacklisted hosts are hosts that are denied access, while whitelisted hosts are granted access. Greylisted hosts, as the name implies, is a kind of cross between blacklisting and whitelisting: such hosts are temporarily blocked (or, in some cases, temporarily allowed) until an additional step is performed.

When spamd receives e-mail from an unknown host, it records three pieces of data (known as a triplet) for each incoming mail message:

  • The IP address of the host
  • The envelope sender address
  • The envelope recipient address (or just the first of them)

With this data, we follow a basic rule: if we have never seen a triplet before, then we refuse delivery and any others that may come within a certain period of time with a temporary failure.

This data s registered on the mail server’s internal database, along with the time stamp of its first appearance. The e-mail message will be dismissed until the configured period of time is elapsed (in the case of spamd, the default time period is 25 minutes). Temporary errors are defined in the Simple Mail Transfer Protocol (SMTP) as 4xx reply codes. Since SMTP is considered an unreliable transport, the possibility of temporary failures is built into the core spec. Therefore, fully capable SMTP implementations are expected to maintain queues for retrying message transmissions in such cases. During the initial testing of greylisting, it was observed that the vast majority of spam appears to be sent from applications designed specifically for spamming. These applications appear to adopt the “fire-and-forget” methodology, and they attempt to send the spam to one or several hosts for a domain, but never attempt a true retry as a real message transfer agent (MTA) would. Conversely, spamd assumes that if a sender has proven itself able to properly retry delivery, then it is a legitimate sender and not a spammer. Thus, such a host will be whitelisted for a longer period of time, so that future delivery attempts will be unimpeded. As an example, spamd can be configured to require a successful delivery attempt against a registered triplet to be no earlier than 25 minutes after registration and not later than 4 hours after it. Repeated delivery attempts before the 25 minute period will be ignored with the same 4xx reply code. After 4 hours the triplet will be expired, so delivery attempts will register anew. When spamd sees an attempt within the 25 minute – 4 hour window, the connecting host will be whitelisted for a certain period of time (default is 36 days).


In addition to being highly effective at blocking spam (over 95% in initial tests), greylisting also ensures that no legitimate mail is ever blocked. In addition, greylisting has been shown to be extremely effective in blocking e-mail-based viruses, as they also do not tend to retry deliveries. And since these viruses are fairly large, bandwidth and processing savings are considerable as compared to the standard method of accepting delivery and local virus scanning.

This temporary rejection can be issued at different stages of the SMTP dialogue, so that the spam deferral daemon can store more or less data about the incoming message. In addition to whitelisting good senders, a greylisting daemon can provide for exceptions. Greylisting can usually be overridden by a fully validated TLS connection with a matching certificate. In addition, because large senders often have a pool of machines that can send and resend e-mail, IP addresses that have the most-significant 24 bits the same are treated as equivalent. In some cases, SPF records are used to determine the sending pool.

Greylisting Disadvantages

Now that we have presented an overview of greylisting and its advantages, it is time to examine some of the disadvantages of this method.

The biggest disadvantage of greylisting is that for unrecognized servers, it destroys the near-instantaneous nature of e-mail most users have come to expect. Mail from unrecognized servers is typically delayed by 15 minutes, and could be delayed for days. A customer of a greylisted ISP cannot always rely on getting e-mail in a predetermined period of time. This disadvantage is mitigated somewhat by the fact that prompt mail delivery is restored once a server has been recognized and is generally maintained automatically so long as users continue to exchange messages. This disadvantage is apparent when a server accessing a greylising mailserver attempts to reset his credentials to a website that uses e-mail confirmation of password resets. In some cases, the delivery delay imposed by the greylister can exceed the expiry time of the password reset token delivered in the e-mail. In such cases, manual intervention may be required to whitelist the host so the e-mail with the reset token can be used before it expires.

Another problem is that some SMTP senders may interpret the temporary rejection as a permanent failure. Old clients conforming only to the obsolete SMTP specification and ignoring its recommendations may give up on delivery after the first failed attempt, as RFC 821 (the old specification) states that clients “should” retry messages rather than using the word “must”. The problem can affect SMTP clients in unexpected ways. Most MTAs will queue and retry messages, but some do not. A possible solution is to configure the application to use a local SMTP server as an outbound queue, instead of attempting direct delivery. The server operator who employs greylisting can whitelist a client which is known to fail on temporary errors.

Some MTAs, upon encountering the temporary failure message from a greylisting server on the first attempt, will send a warning message back to the original sender of the message. The warning message is not a bounce message, but it is often formatted similarly to one and reads like one. This practice often causes the sender to believe the message has not been delivered, although the message will be delivered successfully, albeit at a later time.

Another problem is that the retry might come from a different IP address than the original attempt. When the source of an e-mail is a server farm or goes out through a relay, it is likely that a server other than the original one will make the next attempt. Moreover, their IPs can belong to completely unrelated address blocks (due to network fault tolerance), thereby defying the simple technique of identifying the most significant part of the address. Since the IP addresses will be different, the recipient’s server will fail to recognize that a series of attempts are related, and refuse each of them in turn. This problem can be partially solved by proactively identifying such server farms and making exceptions for hem. Likewise, exceptions have to be configured for multi-homed hosts and hosts using DHCP.


External Links:

The Next Step in the Spam Control War: Greylisting by Evan Harris on projects.puremagic.com

Spam Blocking in BSD with spamd: Part Three (Greylisting)

greylistingIn part one and part two of this series, I discussed the advantages of using spamd and some basic configuration steps. In this article, I will cover greylisting in more detail.

The Problem

Spammers try to use other people’s equipment to send their messages. In addition, the software they install without the legal owner’s permission needs to be lightweight in order to run undetected. Moreover, spammers do not typically consider any individual message they send to be important. From this we can conjecture that the typical spam sender is probably not set up to interpret SMTP status codes correctly, if at all.

The current standard for Internet e-mail transmission is defined in RFC 2821. Section 4.5.4 covers retry strategies, and subsection 4.5.4.1 covers sending strategies. This section specifies the following:

  1. Mail that cannon be transmitted MUST be queued and periodically retried by the sender.
  2. The sender MUST delay retrying a particular destination after one attempt has failed. In general, the retry interval SHOULD be at least 30 minutes; however, more sophisticated and variable strategies will be beneficial when the SMTP clients can determine the reason for non-delivery.
  3. Retries continue until the message is transmitted or the sender gives up; the give-up time generally needs to be at least 4-5 days. The parameters to the retry algorithm MUST be configurable.


The Solution

We can use these requirements to our advantage. When a compromised machine is used to send spam, the sender application tends to try delivery only once, without checking for any results or return codes. Real SMTP implementations interpret SMTP return codes and act on them, as explained above. E-mail, like most other Internet services, are described as best-effort services, with a significant amount of design and development being put into make services such as SMTP fault tolerant. Thus, real mail servers retry if the initial attempt fails with any kind of temporary error.
This is what is so elegant about greylisting. spamd reports a temporary local problem to the client. Well-behaved senders with legitimate messages will try to resend the message later, but spammers have no interest in waiting for the chance to retry, since it will increase their cost of delivering the messages. Thus, the spammers will keep hammering spamd, and will get greylisted.


Configuring spamd for Greylisting

Setting up spamd for greylisting is fairly easy, although under FreeBSD, you need to have a file descriptor filesystem mounted on /dev/fd. You can do this by adding this line to your /etc/fstab file (fstab lists all the available static disks and disk partitions and where they are to be mounted):

fdescfs /dev/fd fdescfs rw 0 0

You also want to place the relevant lines in /etc/rc.conf or /etc/rc.conf.local; e.g.:

spamd_flags = “-v -G 2:4:864”
spamd_grey = YES

This tells spamd to enable vpopmail configuration (-v); -G sets the passtime for greylisting to 2 minutes, greyexp to 4 hours and whiteexp to 864 hours (36 days). This means that the client must wait 2 minutes before resending an e-mail to avoid getting greylisted. A client will remain on the the greylist for 4 hours after violating this standard. If the client waits 2 minutes before resending, it will get whitelisted, and remain on the whitelist for 864 hours.

Why It Might Not Work

The reason why greylisting should work is because any standards compliant SMTP client is required to retry delivery after some reasonable amount of time. But there are circumstances in which it may not work. First of all, the first e-mail message sent from any site which has not contacted you for as long as the greylister keeps its data aroun will be delayed for some random amount of time which depends mainly on the sender’s retry interval. There are some circumstances where avoiding even a minimal delay is desirable. In addition, you are bound to encounter misconfigured mail servers which either do not retry at all or retry at all or retry too quickly, perhaps stopping delivery retries after a few attempts. Finally, there are some sites which are large enough to have several outgoing SMTP servers (e.g. GMail); these sites do not work well with greylisting since they are not guaranteed to retry delivery of any given message from the same IP address as the last delivery attempt for that message. The RFCs do not state that they new delivery attempts have to come from the same IP address, so these sites can legitimately claim to comply with the RFCs.

If you need to make allowances for such situations in your setup, it is fairly easy. One approach is to define a table for a local whitelist; the list should be fed from a text file in case of reboots:

table <my_localwhite> file “/etc/mail/local_whitelist.txt”

To make sure SMTP traffic from the addresses in the table is not fed to spamd, you add a no rdr rule at the top of your redirection block:

no rdr proto tcp from <my_localwhite> to $mailservers port smtp

Once you have added these changes to your rule set, enter the addresses you need to protect from redirection into the local_whitelist.txt file, then reload your rule set using pfctl -f.

External Links:

Greylisting on Wikipedia

RFC 2821 – the RFC with the specifications for Simple Mail Transfer Protocol (SMTP).

Greylisting.org – a site devoted to articles and tools related to greylisting.

Greylisting: The Next Step in the Spam Control War – Greylisting site created by Evan Harris, the guy who originally came up with the idea of greylisting. Also contains whitepapers and a fully functional example of greylisting called relaydelay.

 

 

© 2013 David Zientara. All rights reserved. Privacy Policy