Greylisting Advantages and Disadvantages

greylistingIn the previous two articles, we covered installation and configuration of spamd, a useful spam-referral daemon. In this article, we will examine some of the advantages and disadvantages of greylisting.

The Greylisting Process

Before we begin, it might be useful to review some of the basic concepts of spam deferral. The process involves, at its most basic level, dividing hosts into three categories: blacklisted hosts, whitelisted hosts, and greylisted hosts. Blacklisted hosts are hosts that are denied access, while whitelisted hosts are granted access. Greylisted hosts, as the name implies, is a kind of cross between blacklisting and whitelisting: such hosts are temporarily blocked (or, in some cases, temporarily allowed) until an additional step is performed.

When spamd receives e-mail from an unknown host, it records three pieces of data (known as a triplet) for each incoming mail message:

  • The IP address of the host
  • The envelope sender address
  • The envelope recipient address (or just the first of them)

With this data, we follow a basic rule: if we have never seen a triplet before, then we refuse delivery and any others that may come within a certain period of time with a temporary failure.

This data s registered on the mail server’s internal database, along with the time stamp of its first appearance. The e-mail message will be dismissed until the configured period of time is elapsed (in the case of spamd, the default time period is 25 minutes). Temporary errors are defined in the Simple Mail Transfer Protocol (SMTP) as 4xx reply codes. Since SMTP is considered an unreliable transport, the possibility of temporary failures is built into the core spec. Therefore, fully capable SMTP implementations are expected to maintain queues for retrying message transmissions in such cases. During the initial testing of greylisting, it was observed that the vast majority of spam appears to be sent from applications designed specifically for spamming. These applications appear to adopt the “fire-and-forget” methodology, and they attempt to send the spam to one or several hosts for a domain, but never attempt a true retry as a real message transfer agent (MTA) would. Conversely, spamd assumes that if a sender has proven itself able to properly retry delivery, then it is a legitimate sender and not a spammer. Thus, such a host will be whitelisted for a longer period of time, so that future delivery attempts will be unimpeded. As an example, spamd can be configured to require a successful delivery attempt against a registered triplet to be no earlier than 25 minutes after registration and not later than 4 hours after it. Repeated delivery attempts before the 25 minute period will be ignored with the same 4xx reply code. After 4 hours the triplet will be expired, so delivery attempts will register anew. When spamd sees an attempt within the 25 minute – 4 hour window, the connecting host will be whitelisted for a certain period of time (default is 36 days).


In addition to being highly effective at blocking spam (over 95% in initial tests), greylisting also ensures that no legitimate mail is ever blocked. In addition, greylisting has been shown to be extremely effective in blocking e-mail-based viruses, as they also do not tend to retry deliveries. And since these viruses are fairly large, bandwidth and processing savings are considerable as compared to the standard method of accepting delivery and local virus scanning.

This temporary rejection can be issued at different stages of the SMTP dialogue, so that the spam deferral daemon can store more or less data about the incoming message. In addition to whitelisting good senders, a greylisting daemon can provide for exceptions. Greylisting can usually be overridden by a fully validated TLS connection with a matching certificate. In addition, because large senders often have a pool of machines that can send and resend e-mail, IP addresses that have the most-significant 24 bits the same are treated as equivalent. In some cases, SPF records are used to determine the sending pool.

Greylisting Disadvantages

Now that we have presented an overview of greylisting and its advantages, it is time to examine some of the disadvantages of this method.

The biggest disadvantage of greylisting is that for unrecognized servers, it destroys the near-instantaneous nature of e-mail most users have come to expect. Mail from unrecognized servers is typically delayed by 15 minutes, and could be delayed for days. A customer of a greylisted ISP cannot always rely on getting e-mail in a predetermined period of time. This disadvantage is mitigated somewhat by the fact that prompt mail delivery is restored once a server has been recognized and is generally maintained automatically so long as users continue to exchange messages. This disadvantage is apparent when a server accessing a greylising mailserver attempts to reset his credentials to a website that uses e-mail confirmation of password resets. In some cases, the delivery delay imposed by the greylister can exceed the expiry time of the password reset token delivered in the e-mail. In such cases, manual intervention may be required to whitelist the host so the e-mail with the reset token can be used before it expires.

Another problem is that some SMTP senders may interpret the temporary rejection as a permanent failure. Old clients conforming only to the obsolete SMTP specification and ignoring its recommendations may give up on delivery after the first failed attempt, as RFC 821 (the old specification) states that clients “should” retry messages rather than using the word “must”. The problem can affect SMTP clients in unexpected ways. Most MTAs will queue and retry messages, but some do not. A possible solution is to configure the application to use a local SMTP server as an outbound queue, instead of attempting direct delivery. The server operator who employs greylisting can whitelist a client which is known to fail on temporary errors.

Some MTAs, upon encountering the temporary failure message from a greylisting server on the first attempt, will send a warning message back to the original sender of the message. The warning message is not a bounce message, but it is often formatted similarly to one and reads like one. This practice often causes the sender to believe the message has not been delivered, although the message will be delivered successfully, albeit at a later time.

Another problem is that the retry might come from a different IP address than the original attempt. When the source of an e-mail is a server farm or goes out through a relay, it is likely that a server other than the original one will make the next attempt. Moreover, their IPs can belong to completely unrelated address blocks (due to network fault tolerance), thereby defying the simple technique of identifying the most significant part of the address. Since the IP addresses will be different, the recipient’s server will fail to recognize that a series of attempts are related, and refuse each of them in turn. This problem can be partially solved by proactively identifying such server farms and making exceptions for hem. Likewise, exceptions have to be configured for multi-homed hosts and hosts using DHCP.


External Links:

The Next Step in the Spam Control War: Greylisting by Evan Harris on projects.puremagic.com

spamd: Part One

spamd

The spamd settings page in pfSense.

spamd is a ISC-licensed lightweight spam-deferral daemon which is part of the OpenBSD project. It works directly with SMTP connections and supports such features as greylisting and minimizing false positives. It should be fully functional on any system where pf is available; conveniently, there is a package available for pfSense.

spamd can be used to prevent inbound spam from reaching mail servers. It can also be used as an application-level proxy to ensure that external mail servers connecting to internal mail servers behave legitimately, and it can prevent outgoing spam from systems that may be compromised or under the control of spammers. It can be used to block spammers with the following features:

  • Blacklisting, such as the SPEWS database or other lists of IPs
  • Tarpitting, which can slow down spam reception considerably and hold connections open for a significant amount of time.
  • Greylisting, which forces e-mail to be delayed for a configurable period, requiring the remote end to resend mail at least once in order for it to be delivered.
  • SpamTrapping/GreyTrapping, which is the seeding of an e-mail address so that spammers can find it, but normal users can’t. If the e-mail address is used, then the sender must be a spammer, and they are blacklisted.

spamd Installation and Configuration

Installation of spamd in pfSense is easy. Navigate to System -> Packages, and scroll down to “spamd” in the package list. Click on the “plus” button to select the spamd package, and press the “Confirm” button on the subsequent page to confirm installation, which should complete in about three minutes.

Once spamd is installed, there will be a new item on the “Services” menu called “SpamD“. Navigate to Services -> SpamD to begin spamd configuration. There are four tabs available: “External Sources“, which allows spamd to connect to an external database, “SpamD Whitelist“, which enables you to specify hosts that will be exempt for spamd, “SpamD Settings“, which allows you to configure some general settings, and “SpamD Database“, which produces a listing of items in spamd’s database. We will begin with “SpamD Settings“.


The first setting is “Identifier“. In this edit box, you can specify the SMTP version banner that is reported upon initial connection. At the spamd command line, this would be done with: spamd -n name (where name is the SMTP version banner). The next setting is “Maximum blacklisted connections“. This is the maximum number of concurrent blacklisted connections to allow in greylisting mode. This value may not be greater than the “Max concurrent connections“, which is the next setting. The default is maxcon-100. At the command line, this would be specified with: spamd -B maxblack. You can specify the maximum number of concurrent connections at the command line with: spamd -c maxcon.

The next setting is the “Grey listing” check box. If this box is checked, connections from addresses not blacklisted on the black lists tab will be considered for greylisting. Such connections will receive an SMTP temporary failure message. If the host returns after a certain interval (the default is 25 minutes), the host will be whitelisted. If the host does not wait, however, they will be blacklisted. This is based on the assumption that non-spammers will abide by SMTP protocol practices and wait to re-send a message, while spammers will not play nice, and will likely keep hammering the e-mail server. The next setting, “Passtime“, allows you to adjust the pasttime parameter. After passtime minutes, if spamd sees a retried attempt to deliver mail for the same tuple, spamd will whitelist the connecting address by adding it as a whitelist entry. “Grey Expiration” allows you to set the interval after which spamd removes connection entries from the database if delivery has not been retired within greyexp hours from the initial time a connection is seen. Finally, “White Exp” is the interval after which spamd removes whitelist entries from the database if no mail delivery activity has been seen from the whitelisted address within whitexp hours from the initial time an address is whitelisted.

These four settings (Grey listing, Pasttime, Grey Expiration and White Exp) can be controlled from the spamd command line with a single parameter: spamd -G passtime:greyexp:whitexp. passtime defaults to 25 minutes, greyexp to 4 hours, and white exp to 864 hours (36 days).

In the next article, we will cover the remaining spamd settings, including configuring an external database and whitelisting hosts.


External Links:

OpenBSD’s spamd man page.

Hitting back at spammers with OpenBSD and spamd at www.hungryhacker.com

© 2013 David Zientara. All rights reserved. Privacy Policy