Greylisting Advantages and Disadvantages

greylistingIn the previous two articles, we covered installation and configuration of spamd, a useful spam-referral daemon. In this article, we will examine some of the advantages and disadvantages of greylisting.

The Greylisting Process

Before we begin, it might be useful to review some of the basic concepts of spam deferral. The process involves, at its most basic level, dividing hosts into three categories: blacklisted hosts, whitelisted hosts, and greylisted hosts. Blacklisted hosts are hosts that are denied access, while whitelisted hosts are granted access. Greylisted hosts, as the name implies, is a kind of cross between blacklisting and whitelisting: such hosts are temporarily blocked (or, in some cases, temporarily allowed) until an additional step is performed.

When spamd receives e-mail from an unknown host, it records three pieces of data (known as a triplet) for each incoming mail message:

  • The IP address of the host
  • The envelope sender address
  • The envelope recipient address (or just the first of them)

With this data, we follow a basic rule: if we have never seen a triplet before, then we refuse delivery and any others that may come within a certain period of time with a temporary failure.

This data s registered on the mail server’s internal database, along with the time stamp of its first appearance. The e-mail message will be dismissed until the configured period of time is elapsed (in the case of spamd, the default time period is 25 minutes). Temporary errors are defined in the Simple Mail Transfer Protocol (SMTP) as 4xx reply codes. Since SMTP is considered an unreliable transport, the possibility of temporary failures is built into the core spec. Therefore, fully capable SMTP implementations are expected to maintain queues for retrying message transmissions in such cases. During the initial testing of greylisting, it was observed that the vast majority of spam appears to be sent from applications designed specifically for spamming. These applications appear to adopt the “fire-and-forget” methodology, and they attempt to send the spam to one or several hosts for a domain, but never attempt a true retry as a real message transfer agent (MTA) would. Conversely, spamd assumes that if a sender has proven itself able to properly retry delivery, then it is a legitimate sender and not a spammer. Thus, such a host will be whitelisted for a longer period of time, so that future delivery attempts will be unimpeded. As an example, spamd can be configured to require a successful delivery attempt against a registered triplet to be no earlier than 25 minutes after registration and not later than 4 hours after it. Repeated delivery attempts before the 25 minute period will be ignored with the same 4xx reply code. After 4 hours the triplet will be expired, so delivery attempts will register anew. When spamd sees an attempt within the 25 minute – 4 hour window, the connecting host will be whitelisted for a certain period of time (default is 36 days).


In addition to being highly effective at blocking spam (over 95% in initial tests), greylisting also ensures that no legitimate mail is ever blocked. In addition, greylisting has been shown to be extremely effective in blocking e-mail-based viruses, as they also do not tend to retry deliveries. And since these viruses are fairly large, bandwidth and processing savings are considerable as compared to the standard method of accepting delivery and local virus scanning.

This temporary rejection can be issued at different stages of the SMTP dialogue, so that the spam deferral daemon can store more or less data about the incoming message. In addition to whitelisting good senders, a greylisting daemon can provide for exceptions. Greylisting can usually be overridden by a fully validated TLS connection with a matching certificate. In addition, because large senders often have a pool of machines that can send and resend e-mail, IP addresses that have the most-significant 24 bits the same are treated as equivalent. In some cases, SPF records are used to determine the sending pool.

Greylisting Disadvantages

Now that we have presented an overview of greylisting and its advantages, it is time to examine some of the disadvantages of this method.

The biggest disadvantage of greylisting is that for unrecognized servers, it destroys the near-instantaneous nature of e-mail most users have come to expect. Mail from unrecognized servers is typically delayed by 15 minutes, and could be delayed for days. A customer of a greylisted ISP cannot always rely on getting e-mail in a predetermined period of time. This disadvantage is mitigated somewhat by the fact that prompt mail delivery is restored once a server has been recognized and is generally maintained automatically so long as users continue to exchange messages. This disadvantage is apparent when a server accessing a greylising mailserver attempts to reset his credentials to a website that uses e-mail confirmation of password resets. In some cases, the delivery delay imposed by the greylister can exceed the expiry time of the password reset token delivered in the e-mail. In such cases, manual intervention may be required to whitelist the host so the e-mail with the reset token can be used before it expires.

Another problem is that some SMTP senders may interpret the temporary rejection as a permanent failure. Old clients conforming only to the obsolete SMTP specification and ignoring its recommendations may give up on delivery after the first failed attempt, as RFC 821 (the old specification) states that clients “should” retry messages rather than using the word “must”. The problem can affect SMTP clients in unexpected ways. Most MTAs will queue and retry messages, but some do not. A possible solution is to configure the application to use a local SMTP server as an outbound queue, instead of attempting direct delivery. The server operator who employs greylisting can whitelist a client which is known to fail on temporary errors.

Some MTAs, upon encountering the temporary failure message from a greylisting server on the first attempt, will send a warning message back to the original sender of the message. The warning message is not a bounce message, but it is often formatted similarly to one and reads like one. This practice often causes the sender to believe the message has not been delivered, although the message will be delivered successfully, albeit at a later time.

Another problem is that the retry might come from a different IP address than the original attempt. When the source of an e-mail is a server farm or goes out through a relay, it is likely that a server other than the original one will make the next attempt. Moreover, their IPs can belong to completely unrelated address blocks (due to network fault tolerance), thereby defying the simple technique of identifying the most significant part of the address. Since the IP addresses will be different, the recipient’s server will fail to recognize that a series of attempts are related, and refuse each of them in turn. This problem can be partially solved by proactively identifying such server farms and making exceptions for hem. Likewise, exceptions have to be configured for multi-homed hosts and hosts using DHCP.


External Links:

The Next Step in the Spam Control War: Greylisting by Evan Harris on projects.puremagic.com

spamd: Part One

spamd

The spamd settings page in pfSense.

spamd is a ISC-licensed lightweight spam-deferral daemon which is part of the OpenBSD project. It works directly with SMTP connections and supports such features as greylisting and minimizing false positives. It should be fully functional on any system where pf is available; conveniently, there is a package available for pfSense.

spamd can be used to prevent inbound spam from reaching mail servers. It can also be used as an application-level proxy to ensure that external mail servers connecting to internal mail servers behave legitimately, and it can prevent outgoing spam from systems that may be compromised or under the control of spammers. It can be used to block spammers with the following features:

  • Blacklisting, such as the SPEWS database or other lists of IPs
  • Tarpitting, which can slow down spam reception considerably and hold connections open for a significant amount of time.
  • Greylisting, which forces e-mail to be delayed for a configurable period, requiring the remote end to resend mail at least once in order for it to be delivered.
  • SpamTrapping/GreyTrapping, which is the seeding of an e-mail address so that spammers can find it, but normal users can’t. If the e-mail address is used, then the sender must be a spammer, and they are blacklisted.

spamd Installation and Configuration

Installation of spamd in pfSense is easy. Navigate to System -> Packages, and scroll down to “spamd” in the package list. Click on the “plus” button to select the spamd package, and press the “Confirm” button on the subsequent page to confirm installation, which should complete in about three minutes.

Once spamd is installed, there will be a new item on the “Services” menu called “SpamD“. Navigate to Services -> SpamD to begin spamd configuration. There are four tabs available: “External Sources“, which allows spamd to connect to an external database, “SpamD Whitelist“, which enables you to specify hosts that will be exempt for spamd, “SpamD Settings“, which allows you to configure some general settings, and “SpamD Database“, which produces a listing of items in spamd’s database. We will begin with “SpamD Settings“.


The first setting is “Identifier“. In this edit box, you can specify the SMTP version banner that is reported upon initial connection. At the spamd command line, this would be done with: spamd -n name (where name is the SMTP version banner). The next setting is “Maximum blacklisted connections“. This is the maximum number of concurrent blacklisted connections to allow in greylisting mode. This value may not be greater than the “Max concurrent connections“, which is the next setting. The default is maxcon-100. At the command line, this would be specified with: spamd -B maxblack. You can specify the maximum number of concurrent connections at the command line with: spamd -c maxcon.

The next setting is the “Grey listing” check box. If this box is checked, connections from addresses not blacklisted on the black lists tab will be considered for greylisting. Such connections will receive an SMTP temporary failure message. If the host returns after a certain interval (the default is 25 minutes), the host will be whitelisted. If the host does not wait, however, they will be blacklisted. This is based on the assumption that non-spammers will abide by SMTP protocol practices and wait to re-send a message, while spammers will not play nice, and will likely keep hammering the e-mail server. The next setting, “Passtime“, allows you to adjust the pasttime parameter. After passtime minutes, if spamd sees a retried attempt to deliver mail for the same tuple, spamd will whitelist the connecting address by adding it as a whitelist entry. “Grey Expiration” allows you to set the interval after which spamd removes connection entries from the database if delivery has not been retired within greyexp hours from the initial time a connection is seen. Finally, “White Exp” is the interval after which spamd removes whitelist entries from the database if no mail delivery activity has been seen from the whitelisted address within whitexp hours from the initial time an address is whitelisted.

These four settings (Grey listing, Pasttime, Grey Expiration and White Exp) can be controlled from the spamd command line with a single parameter: spamd -G passtime:greyexp:whitexp. passtime defaults to 25 minutes, greyexp to 4 hours, and white exp to 864 hours (36 days).

In the next article, we will cover the remaining spamd settings, including configuring an external database and whitelisting hosts.


External Links:

OpenBSD’s spamd man page.

Hitting back at spammers with OpenBSD and spamd at www.hungryhacker.com

Spam Blocking in BSD: Part Four (spamlogd and spamdb)

spamlogdIn parts one, two and three, I discussed the spamd daemon, how to configure it, and how to set it up for greylisting. In this part, I will discuss spamlogd and spamdb.

Using spamlogd

Spamlogd is a whitelist updater, and it works quietly in the background, recording logged connections to and from your mail servers to keep your whitelist updated. It thus manipulates the spamd database in /var/db/spamd used for greylisting. Spamlogd helps ensure valid e-mail to and from hosts you communicate with regularly goes through with a minimum of fuss.

In order to perform its job properly, spamlogd needs you to log SMTP connections to and from your mail servers (this will enable it to add IP addresses that receive e-mail from you to the whitelist). On OpenBSD 4.1 and later, you can create several pflog interfaces and specify which interface a rule should log to. If you want to separate the data spamlogd needs to read from the rest of your PF logs, create a separate pflog1 interface using this command:

ifconfig pflog1 create

Alternatively, you can create a hostname.pflog1 file that contains only the line “up”. We want to add this to the rules, or if you already have rules for logging, change them accordingly:

pass log (to pflog1) proto tcp from any to $emailserver port $email

pass log (to pflog1) proto tcp from $emailserver to any port smtp


Where $emailserver is the IP address of the e-mail server and $email is the port the server is listening on. Next you need to add:

-l pflog1

to spamlogd’s setup parameters. Now you have separated the spamd-related logging from the rest of the logging. Now, spamlogd will add the IP addresses that receive e-mail you send to the whitelist.

Using spamdb

If you need to view or change the contents of your blacklists, whitelists or greylists, then spamdb should be your main interface to managing the lists. From the earliest times, spamdb offered the capability of adding whitelist entries to the database (spamdb -a x.x.x.x) or deleting entries (spamdb -d x.x.x.x). Now, spamdb has even more capabilities, such as the capability of adding a client to a traplist for greytrapping. A traplist is a key component in greylisting; it contains a list of invalid e-mail addresses on servers for which we handle e-mail. When the spammer connects to our server for the first time, he will be greylisted, just like any client who has not connected to the server before. The second time, if the spammer tries to resend the same e-mail without waiting long enough or if he tries to send an e-mail to an address on the traplist, he will be caught in our tarpit and the SMTP header will be sent 1 byte at a time. The command:

sudo spamdb -T -a <e-mail address>

will add the e-mail address specified to the traplist.


External Links:

spamlogd at openbsd.org

spamdb at openbsd.org

Spam Blocking in BSD with spamd: Part Three (Greylisting)

greylistingIn part one and part two of this series, I discussed the advantages of using spamd and some basic configuration steps. In this article, I will cover greylisting in more detail.

The Problem

Spammers try to use other people’s equipment to send their messages. In addition, the software they install without the legal owner’s permission needs to be lightweight in order to run undetected. Moreover, spammers do not typically consider any individual message they send to be important. From this we can conjecture that the typical spam sender is probably not set up to interpret SMTP status codes correctly, if at all.

The current standard for Internet e-mail transmission is defined in RFC 2821. Section 4.5.4 covers retry strategies, and subsection 4.5.4.1 covers sending strategies. This section specifies the following:

  1. Mail that cannon be transmitted MUST be queued and periodically retried by the sender.
  2. The sender MUST delay retrying a particular destination after one attempt has failed. In general, the retry interval SHOULD be at least 30 minutes; however, more sophisticated and variable strategies will be beneficial when the SMTP clients can determine the reason for non-delivery.
  3. Retries continue until the message is transmitted or the sender gives up; the give-up time generally needs to be at least 4-5 days. The parameters to the retry algorithm MUST be configurable.


The Solution

We can use these requirements to our advantage. When a compromised machine is used to send spam, the sender application tends to try delivery only once, without checking for any results or return codes. Real SMTP implementations interpret SMTP return codes and act on them, as explained above. E-mail, like most other Internet services, are described as best-effort services, with a significant amount of design and development being put into make services such as SMTP fault tolerant. Thus, real mail servers retry if the initial attempt fails with any kind of temporary error.
This is what is so elegant about greylisting. spamd reports a temporary local problem to the client. Well-behaved senders with legitimate messages will try to resend the message later, but spammers have no interest in waiting for the chance to retry, since it will increase their cost of delivering the messages. Thus, the spammers will keep hammering spamd, and will get greylisted.


Configuring spamd for Greylisting

Setting up spamd for greylisting is fairly easy, although under FreeBSD, you need to have a file descriptor filesystem mounted on /dev/fd. You can do this by adding this line to your /etc/fstab file (fstab lists all the available static disks and disk partitions and where they are to be mounted):

fdescfs /dev/fd fdescfs rw 0 0

You also want to place the relevant lines in /etc/rc.conf or /etc/rc.conf.local; e.g.:

spamd_flags = “-v -G 2:4:864”
spamd_grey = YES

This tells spamd to enable vpopmail configuration (-v); -G sets the passtime for greylisting to 2 minutes, greyexp to 4 hours and whiteexp to 864 hours (36 days). This means that the client must wait 2 minutes before resending an e-mail to avoid getting greylisted. A client will remain on the the greylist for 4 hours after violating this standard. If the client waits 2 minutes before resending, it will get whitelisted, and remain on the whitelist for 864 hours.

Why It Might Not Work

The reason why greylisting should work is because any standards compliant SMTP client is required to retry delivery after some reasonable amount of time. But there are circumstances in which it may not work. First of all, the first e-mail message sent from any site which has not contacted you for as long as the greylister keeps its data aroun will be delayed for some random amount of time which depends mainly on the sender’s retry interval. There are some circumstances where avoiding even a minimal delay is desirable. In addition, you are bound to encounter misconfigured mail servers which either do not retry at all or retry at all or retry too quickly, perhaps stopping delivery retries after a few attempts. Finally, there are some sites which are large enough to have several outgoing SMTP servers (e.g. GMail); these sites do not work well with greylisting since they are not guaranteed to retry delivery of any given message from the same IP address as the last delivery attempt for that message. The RFCs do not state that they new delivery attempts have to come from the same IP address, so these sites can legitimately claim to comply with the RFCs.

If you need to make allowances for such situations in your setup, it is fairly easy. One approach is to define a table for a local whitelist; the list should be fed from a text file in case of reboots:

table <my_localwhite> file “/etc/mail/local_whitelist.txt”

To make sure SMTP traffic from the addresses in the table is not fed to spamd, you add a no rdr rule at the top of your redirection block:

no rdr proto tcp from <my_localwhite> to $mailservers port smtp

Once you have added these changes to your rule set, enter the addresses you need to protect from redirection into the local_whitelist.txt file, then reload your rule set using pfctl -f.

External Links:

Greylisting on Wikipedia

RFC 2821 – the RFC with the specifications for Simple Mail Transfer Protocol (SMTP).

Greylisting.org – a site devoted to articles and tools related to greylisting.

Greylisting: The Next Step in the Spam Control War – Greylisting site created by Evan Harris, the guy who originally came up with the idea of greylisting. Also contains whitepapers and a fully functional example of greylisting called relaydelay.

 

 

Spam Blocking in BSD with spamd: Part Two

spamdIn part one of this series, I discussed some of the advantages of spamd and how it works. In this article, I will cover spamd configuration.

spamd Configuration

Configuring the spamd daemon is pretty straightforward. First, you have to edit your rule set in /etc/pf.conf to include the following:

table persist
table persist
 pass on $ext_if inet proto tcp from <spamd> to \
( $ext_if, $localnet ) smtp rdr-to 127.0.0.1 port 8025
pass on $ext_if inet proto tcp from \

!<spamd-white> to ( $ext_if, $localnet ) port smtp rdr-to 127.0.0.1 port 8025

These rules set up two tables, (a table of known spammers) and (a list of whitelisted hosts which we assume are not spammers). SMTP traffic from the addresses in the first table () plus the ones not in the second table () is redirected to a daemon listening at port 8025 of the machine. The application which uses these tables, of course, is spamd, which is designed to waste spammers’ time and keep their traffic off our network.


You also need to edit your spamd.conf file (under FreeBSD. it should be located at /usr/local/etc/spamd/spamd.conf). First, you need to define which lists you will use:

all:\
:my_blacklist:my_whitelist:

This is where you add all the blacklists you want to use, separated by colons (:). If you want to use whitelists to subtract addresses from your blacklist, you add the name of the whitelist immediately after the name of each blacklist.

Next, you need to define your blacklist:

my_blacklist:\
:black:\
:msg=”SPAM. Your address %A has sent pam with the last 24 hours”:\
:method=http:\
:file=www.openbsd.org/spamd/traplist.gz

After the name, the first data field specifies the list type (black). The msg field contains the message to display to blacklisted senders during the SMTP dialogue. The method field specifies how spamd-setup fetches the list data. In this example, the type is http, but there are other options: you can fetch the list data via ftp, from a file in a mounted file system or via exec of an exernal program. The file filed specifies the name of the file spamd expects to receive.


You also need to define a whitelist:

my_whitelist:\
:white:\
:method=file:\
:file=/etc/mail/whitelist.txt

The whitelist in this example is similar to the blacklist, only the message parameter is omitted since it is not needed, and the source is a file on the local system.

You should be aware that enabling the suggested blacklists in the default spamd.conf could lead to blacklisting of large blocks of the internet, including several countries such as South Korea. Using other lists than the default ones is possible, and its within your discretion to use other ones or to make your own copy of the default blacklist and edit it as needed.

Next, put the lines for spamd and the startup parameters you want in /etc/rc.conf or /etc/rc.conf.local. In FreeBSD, spamd_grey has to be set to YES to enable greylisting (on OpenBSD 4.1 and later, spamd runs in greylisting mode by default and you need to set spamd_black to YES to run spamd in pure blacklisting mode. There are several parameters that can be used; see the spamd man page for a complete list of these options.

You can then view the table contents using pfctl or other applications. If you want to change or delete entries, you should use the spamdb utility, not pfctl. Also, note that if the redirection (rdr) rules you use do not include a “pass” part, you need to set up pass rules to let through traffic to your redirection, and also set up rules to let legitimate e-mail through.

External Links:

spamd man page at openbsd.org

The Apache SpamAssassin Project – download SpamAssassin here; you can also find SpamAssassin’s documentation.

Spam Blocking in BSD with spamd: Part One

SpamKeeping spam out of the network is one of the more conspicuous responsibilities of the typical network tech. Fortunately, not only is FreeBSD (the OS under which pfSense runs) secure and solid, but tools are available for this platform which should make your life easier.

Spam Blocking: Enter spamd

The main tool for combating spam under BSD is spamd. Spamd is a BSD-licensed lightweight spam-deferral daemon written for OpenBSD. It is included with OpenBSD, and is designed to work in conjunction with pf, and should be fully functional on any POSIX system where pf is available, including NetBSD, FreeBSD, and DragonFly BSD. This daemon’s design is based on the following underlying assumptions:

  • Spammers send a large number of messages, and the probability that you are the first person receiving a particular message is small.
  • Spam is sent mainly by a few spammer-friendly networks and a large number of hijacked machines.


Both the messages and the machines will be reported to blacklists quickly, and you can use these blacklists as the basis of your own blacklist. When a blacklisted computer makes an SMTP connection to your e-mail server, spamd presents its banner and immediately switches to a mode where it answers SMTP traffic 1 byte at a time. If the sender is patient enough to complete the SMTP dialogue, which will take ten minutes or more, spamd will return a “temporary error” code (4xx), which indicates that the message could not be delivered successfully and that the sender should keep the mail in his queue and retry later. If he does, the same procedure repeats, until he gives up, which should waste both his queue space and socket handles for several days. Such a process is called tarpitting, and it has the potential to tie up the spammer’s resources to a significant degree at a minimal cost to the system defending against such an attack.


spamd can be used quite effectively as a traditional spam filter, with a blacklist and a whitelist. But it also supports greylisting, which works by rejecting messages from unknown hosts temporarily with 4xx reply codes (temporary errors). Fully capable SMTP implementations are expected to maintain queues for retrying message transmissions in such cases (this behavior is required if an SMTP sender is to comply with the relevant RFCs). When a sender has proven itself able to properly retry delivery, it will be whitelisted for a longer period of time, so that future delivery attempts will be unimpeded. The viability of greylisting as a tactic for blocking unwanted e-mail was first discussed in a 2003 paper by Evan Harris. It is a simple technique, and amazingly enough, it still works, as spammers and malware writers have been slow to adapt.

External Links:

spamd at Wikipedia

RFC 1123 and RFC 2821 at IETF – the key RFCs concerning SMTP’s fault tolerance capabilities

Giving spammers a hard time – a tutorial

Annoying spammers with pf and spamd

spamd tarpit/greylisting “how to”

© 2013 David Zientara. All rights reserved. Privacy Policy