Web Filtering with SquidGuard: Part One

web filtering

The General settings tab for SquidGuard under pfSense 2.1.3.

Now that we’ve covered both Squid and LightSquid, I thought it might be useful to cover some other Squid plugins. In this article, we will cover how to implement web filtering with SquidGuard.

SquidGuard is a URL redirector software, which can be used for web filtering of sites users can access. It uses blacklists to define sites for which access is redirected. Here, we are concerned mainly with SquidGuard installation under pfSense, but it can also be installed under Unix or GNU/Linux. The software’s filtering extends to all computers in an organization, including Windows, Macintosh, UNIX and Linux computers. It was originally developed by Pål Baltzersen and Lars Erik Håland, and was implemented and extended by Lars Erik Håland in the 1990s. The current version is 1.4, released in 2009.

As with other packages, installation of SquidGuard is easy. Just navigate to System -> Packages, scroll down to SquidGuard on the package list, click on the plus button on the right side of the listing, and on the next screen, click the “Confirm” button to confirm installation. The installation will take a few minutes. Once installation is complete, you will have a new menu item under Services called Proxy Filter, with which web filtering can be implemented.

The basic configuration of SquidGuard can be gleaned from the documentation on the official SquidGuard site. The simplest web filtering configuration has a single list of blocked sites and the URL of the page to show when the user tries to access a blocked site. The administrator, though, may choose to define more than one list, each representing a category to block. Finally, sometimes there is a demand to allow specific URLs and domains although they are part of the blacklists for a good reason. In this case, you want to whitelist these domains and URLs. This is generally accomplish in SquidGuard by editing the squidGuard.conf file, but if SquidGuard is installed under pfSense, the basic configuration can be done from the pfSense web GUI.

Web Filtering with SquidGuard: Configuration

You can configure SquidGuard in pfSense by navigating to Services -> Proxy Filter and clicking on the General settings tab. There is a check box at the top; check this box to enable the blacklist. Also on the General settings tab (towards the bottom of the page), you can specify the URL of the blacklist. You can use one of your own blacklists, or you can use one of the publicly available lists on the web. The official SquidGuard site has one at http://www.squidguard.org/blacklists.html. Enter the URL of the blacklist you want to use for web filtering in the appropriate edit box. Once you have done this, press the “Save” button at the bottom of the page.

web filtering

The Blacklist tab in SquidGuard; here you can download blacklists.

Next, click on the blacklist tab and press the “Download” button. Once the download is finished, the status box will display “Blacklist update complete” (it may take several minutes to download).

Once you have uploaded your blacklist, you will need to configure which categories of sites on the blacklist should be either allowed, blocked, or whitelisted. The simplest way to configure it is with the common ACL tab. The common access list settings will apply to all users of the proxy. The next tab is the “Groups ACL” tab, and if you want to apply different rules to ther source networks you should use this tab. This way, you wan have different policies for different networks. When you modify a target rule, you need to click on “Apply” in the General settings tab in order for the changes made to take effect.

In the dropdown box next to each of the target rules, you can select one of the following three web filtering actions:

  • Allow: Grant access to the target category unless blocked by another rule or exception
  • Deny: Block access to sites in the target category
  • Whitelist: Always allow access to the target category (even if blocked by another rule or exception.

At the bottom of the page, there is a check box to block IP addresses in the URL. This will prevent users from bypassing the filter by using the IP of the web site instead of the URL. They can still use the IP addresses of whitelisted sites in the URL, though.

By default, when a user attempts to visit a blocked page, they will see an internal error page indicating that the page was blocked and under which target category it falls. However, you can change the redirect page to a blank page, or any other internal or external URL.

In the next article, we will continue our look at SquidGuard configuration.

External Links:

The official SquidGuard site

URL Filtering – How To Configure SquidGuard in pfSense on hubpages.com

SquidGuard on Wikipedia

Squid Proxy Configuration in pfSense

Squid proxy

Installing Squid under pfSense 2.1.3.

Squid is a proxy server and web cache daemon. It was originally developed as part of the Harvest project at the University of Colorado Boulder. Further work on the program was completed at the University of California, San Diego (UCSD) and funded via two grants from the National Science Foundation. Duane Wessels forked the last pre-commercial version of Harvest and renamed it Squid, and Squid version 1.0.0 was released in July 1996. It has a number of uses. Under pfSense, it can be used to cache repeated requests.

Squid Proxy Installation

Installing and configuring a Squid proxy server under pfSense is relatively easy. From the pfSense web GUI, go to the top menu and navigate to System -> Preferences. Scroll down to “squid” on the list of packages, and click on the “plus” button on the right to install Squid. On the next screen, press the “Confirm” button to confirm that you want to install Squid. It will take a few minutes for the package installer to unpack and install Squid.

Squid Proxy Configuration

Once installation is complete, “Proxy Server” will show up as an option under Services. Navigate to Services -> Proxy Server to configure Squid. Most users will find the default settings to be acceptable, but there are several settings worth noting. There are 7 tabs in the Squid proxy settings: General, Upstream Proxy, Cache Mgmt, Access Control, Traffic Mgmt, Auth Settings, and Local Users.

Squid proxy

The General settings tab in Squid proxy configuration.

General Settings: This covers the most commonly configured Squid proxy settings. The first setting, “Proxy Interface“, determines which interface or interfaces the proxy will bind to. You probably want to leave this set to “LAN” so that the proxy server is accessible to hosts connected. You probably want to leave “Allow users on interface” checked, to allow users connected to the interface selected in the Proxy Interface field to use the proxy. You probably also want to check the “Transparent proxy” check box so all requests for destination port 80 will be forwarded to the proxy server.

Upstream Proxy: If you want Squid to forward requests to an upstream proxy server, you can enable forwarding here. There are also settings to specify the IP address/hostname of the proxy, TCP and ICP port, and username and password, if the upstream proxy requires them.

Cache Mgmt: This controls a number of settings relating to the cache size. “Hard disk cache size” sets the total amount of hard disk space Squid will use to cache objects. If you have a large hard drive, you can increase this setting to cache more objects; otherwise you can probably leave it at the default value (100 MB).

Memory cache size” is the amount of physical RAM to be used for negative cache and in-transit objects. Objects that squid cannot store in memory end up getting swapped to disk which is much slower than RAM. Squid recommends that this setting should be 50% or less of the installed RAM.

Maximum object size” sets the maximum size of objects saved on disk. The default value is 4 KB. You can increase this parameter to save bandwidth, or lower it to improve speed. Most cache hits tend to take place on small files, although you probably want to increase this parameter from the default of 4 KB.

Access Control: This controls a number of settings regarding who is allowed to access the proxy server. In “Allowed subnets“, you can enter each subnet that is allowed to use the proxy (the proxy interface subnet specified in “General” is already an allowed subnet, so you don’t have to specify it here). “Unrestricted IPs” allows you to specify IP addresses that will not be filtered out by the other access control directives set out in this section. “Banned host addresses” allows you to specify IP addresses that are not allowed to use the proxy. “Whitelist” allows you to specify domains that will be accessible to users that are allowed to use the proxy. “Blacklist” allows you to specify domains that will be blocked to users that are allowed to use the proxy.

Traffic Mgmt: This controls a number of traffic settings. “Maximum download size” limits the maximum total download size to the size specified here (the default is no limit). “Maximum upload size” limits the maximum total upload size to the size specified here (the default is no limit). “Overall bandwidth throttling” specifies the bandwidth throttle for downloads; users will gradually have their download speed increased to this value (default is no throttling). “Per host throttling” specifies the download throttling per host (again, the default is no throttling).

Proxy server

The Authentication settings tab.

Auth Settings: Here, you can enable authentication of Squid proxy users. Squid supports local authentication, as well as authentication through an external LDAP, RADIUS or NT server. The default setting is “None” for no authentication.

Local Users: This allows you to set usernames and passwords for individual users. Assuming authentication is enabled and you chose the local authentication option, users will then be able to use the credentials set here to log in to the Squid proxy server.

For basic Squid proxy usage, the above may be all the information you need. If you really want to understand some of the more advanced options, though, you probably should read the Squid man page. You can use these command-line options by [1] executing a command from the Command prompt option in the Diagnostics menu; [2] SSH-ing into your pfSense box; or [3] accessing pfSense directly from the console.

To shut down Squid, issue this command:

squid -k shutdown

To restart squid, issue this command:

/usr/local/sbin/squid -D

However, it should be noted that pfSense seems to start Squid on its own if it notices Squid is not running. Some of the more interesting options are -u port (to specify a different ICP port than 3130; this can also be done from the pfSense GUI), and -z to create swap directories (useful if you have just deleted the cache and want to recreate the swap directories). There is also a loader.conf.local file in the /boot directory with settings that can be configured. The “Squid Package Tuning” document on doc.pfsense.org suggests changing the kern.ipc.nmbclusters parameter from “0” to “32768”. This increases the amount of memory used for socket buffers, and may improve performance.

External Links:

Squid on Wikipedia

How to Set Up a Transparent Squid Proxy Server Using pfSense at hubpages.com

Squid Package Tuning at doc.pfsense.org

Proxy Servers at pfsensesolution.blogspot.com

Spam Blocking in BSD with spamd: Part Two

spamdIn part one of this series, I discussed some of the advantages of spamd and how it works. In this article, I will cover spamd configuration.

spamd Configuration

Configuring the spamd daemon is pretty straightforward. First, you have to edit your rule set in /etc/pf.conf to include the following:

table persist
table persist
 pass on $ext_if inet proto tcp from <spamd> to \
( $ext_if, $localnet ) smtp rdr-to port 8025
pass on $ext_if inet proto tcp from \

!<spamd-white> to ( $ext_if, $localnet ) port smtp rdr-to port 8025

These rules set up two tables, (a table of known spammers) and (a list of whitelisted hosts which we assume are not spammers). SMTP traffic from the addresses in the first table () plus the ones not in the second table () is redirected to a daemon listening at port 8025 of the machine. The application which uses these tables, of course, is spamd, which is designed to waste spammers’ time and keep their traffic off our network.

You also need to edit your spamd.conf file (under FreeBSD. it should be located at /usr/local/etc/spamd/spamd.conf). First, you need to define which lists you will use:


This is where you add all the blacklists you want to use, separated by colons (:). If you want to use whitelists to subtract addresses from your blacklist, you add the name of the whitelist immediately after the name of each blacklist.

Next, you need to define your blacklist:

:msg=”SPAM. Your address %A has sent pam with the last 24 hours”:\

After the name, the first data field specifies the list type (black). The msg field contains the message to display to blacklisted senders during the SMTP dialogue. The method field specifies how spamd-setup fetches the list data. In this example, the type is http, but there are other options: you can fetch the list data via ftp, from a file in a mounted file system or via exec of an exernal program. The file filed specifies the name of the file spamd expects to receive.

You also need to define a whitelist:


The whitelist in this example is similar to the blacklist, only the message parameter is omitted since it is not needed, and the source is a file on the local system.

You should be aware that enabling the suggested blacklists in the default spamd.conf could lead to blacklisting of large blocks of the internet, including several countries such as South Korea. Using other lists than the default ones is possible, and its within your discretion to use other ones or to make your own copy of the default blacklist and edit it as needed.

Next, put the lines for spamd and the startup parameters you want in /etc/rc.conf or /etc/rc.conf.local. In FreeBSD, spamd_grey has to be set to YES to enable greylisting (on OpenBSD 4.1 and later, spamd runs in greylisting mode by default and you need to set spamd_black to YES to run spamd in pure blacklisting mode. There are several parameters that can be used; see the spamd man page for a complete list of these options.

You can then view the table contents using pfctl or other applications. If you want to change or delete entries, you should use the spamdb utility, not pfctl. Also, note that if the redirection (rdr) rules you use do not include a “pass” part, you need to set up pass rules to let through traffic to your redirection, and also set up rules to let legitimate e-mail through.

External Links:

spamd man page at openbsd.org

The Apache SpamAssassin Project – download SpamAssassin here; you can also find SpamAssassin’s documentation.

Spam Blocking in BSD with spamd: Part One

SpamKeeping spam out of the network is one of the more conspicuous responsibilities of the typical network tech. Fortunately, not only is FreeBSD (the OS under which pfSense runs) secure and solid, but tools are available for this platform which should make your life easier.

Spam Blocking: Enter spamd

The main tool for combating spam under BSD is spamd. Spamd is a BSD-licensed lightweight spam-deferral daemon written for OpenBSD. It is included with OpenBSD, and is designed to work in conjunction with pf, and should be fully functional on any POSIX system where pf is available, including NetBSD, FreeBSD, and DragonFly BSD. This daemon’s design is based on the following underlying assumptions:

  • Spammers send a large number of messages, and the probability that you are the first person receiving a particular message is small.
  • Spam is sent mainly by a few spammer-friendly networks and a large number of hijacked machines.

Both the messages and the machines will be reported to blacklists quickly, and you can use these blacklists as the basis of your own blacklist. When a blacklisted computer makes an SMTP connection to your e-mail server, spamd presents its banner and immediately switches to a mode where it answers SMTP traffic 1 byte at a time. If the sender is patient enough to complete the SMTP dialogue, which will take ten minutes or more, spamd will return a “temporary error” code (4xx), which indicates that the message could not be delivered successfully and that the sender should keep the mail in his queue and retry later. If he does, the same procedure repeats, until he gives up, which should waste both his queue space and socket handles for several days. Such a process is called tarpitting, and it has the potential to tie up the spammer’s resources to a significant degree at a minimal cost to the system defending against such an attack.

spamd can be used quite effectively as a traditional spam filter, with a blacklist and a whitelist. But it also supports greylisting, which works by rejecting messages from unknown hosts temporarily with 4xx reply codes (temporary errors). Fully capable SMTP implementations are expected to maintain queues for retrying message transmissions in such cases (this behavior is required if an SMTP sender is to comply with the relevant RFCs). When a sender has proven itself able to properly retry delivery, it will be whitelisted for a longer period of time, so that future delivery attempts will be unimpeded. The viability of greylisting as a tactic for blocking unwanted e-mail was first discussed in a 2003 paper by Evan Harris. It is a simple technique, and amazingly enough, it still works, as spammers and malware writers have been slow to adapt.

External Links:

spamd at Wikipedia

RFC 1123 and RFC 2821 at IETF – the key RFCs concerning SMTP’s fault tolerance capabilities

Giving spammers a hard time – a tutorial

Annoying spammers with pf and spamd

spamd tarpit/greylisting “how to”

© 2013 David Zientara. All rights reserved. Privacy Policy