Reverse Proxy Services with Varnish (Part Three)

reverse proxy

LB Directors tab in Varnish settings under pfSense 2.1.3.

In the last two articles, we introduced the Varnish reverse proxy and covered installation and basic configuration. In this article, we will cover the remaining configuration options.

The third tab on the Varnish settings page is “Custom VCL“. The first two fields are “vcl_recv_early” and “vcl_recv_late“. Code pasted into “vcl_recv_early” will be executed at the beginning of the vcl_recv function and code pasted into “vcl_recv_late” will be executed at the end of the vcl_recv function. vcl_recv is called at the beginning of a request for a document, after the complete request has been received and parsed. Its purpose is to decide whether or not to serve the request, how to do it, and if applicable, which backend to use. You can use these fields to alter the request, if necessary. Typically you can alter the cookies and add and remove request headers.

The next two fields are “vcl_fetch_early” and “vcl_fetch_late“. Code pasted into “vcl_fetch_early” will be included at the beginning of the vcl_fetch function, and code pasted into “vcl_fetch_late” will be included at the end of the vcl_fetch function. vcl_fetch is called after a document has been successfully retrieved from the backend. Normal tasks included here are to alter the response headers, trigger ESI processing, and try alternate backend servers in case the request failed. The last two fields are for “vcl_pipe_early” and “vcl_pipe_late“. Code pasted into “vcl_pipe_early” will be included at the beginning of the vcl_pipe function, and code pasted into “vcl_pipe_late” will be included at the end of the vcl_pipe function. vcl_pipe is called upon entering pipe mode. In this mode, the request is passed on to the backend, and any further data from either client or backend is passed on unaltered until either end closes the connection.


The next tab is “LB Directors“. You can group several backends into a group of backends; these groups are called directors. this will give you increased performance and resilience. Press the “plus” button to add a new director. The first heading is “Director Settings“. The “Director name” is where you can specify a name for the group. In “Match type“, you can select the field type that you would like to use in matching the host or URL, and in the “Host” and “URL” fields, you can specify both of these. The “Rewrite Host” and “Rewrite URL” fields allow you to specify an alternate host and URL that will be redirected to the host and URL specified above. “Req Grace” specifies how long Varnish will keep cached objects for the director. “Additions options” allows you to paste custom Varnish code for this host.

The next subsection is “Backend Settings“. The “Algorithms” dropdown box allows you to select how Varnish will balance clients. “Round-robin” selects a backend in round-robin fashion. “Random selects a backend randomly. “Client” picks a backend based on the client’s identity. You can set the VCL variable client.identity to identify the client by picking up the value of a session cookie or something similar. “Hash” causes Varnish to select a backend based on the URL hash value. This is useful if you are using Varnish to load balance in front of other Varnish caches or other web accelerators as objects won’t be duplicated across caches. “Backend” is where you specify the backends for this director (from the dropdown list) and a weight for each one. The last section, “Failover Settings“, allows you to select a director for failover.

The next tab, “XMLRPC Sync“, allows you to sync Varnish configuration changes. In the first dropdown box, you can choose “Sync to configured system backup server” to sync to a server specified in “High Avail. Sync” under the System menu. “Sync to host(s) defined below” allows you to sync to one or more servers specified at “Remote Server“. “Do not sync this package configuration” disables synchronization. “Sync timeout” allows you to specify a maximum sync wait time.

There are two more tabs. “View Configuration” allows you to view (but not edit) the underlying default.vcl file. “VarnishSTAT” allows you to view the VarnishSTAT server logs.


External Links:

The official Varnish web site

Reverse Proxy Services with Varnish (Part Two)

reverse proxy

Worker thread settings and VCL settings in Varnish under pfSense 2.1.3.

In the last article, we introduced Varnish, a powerful reverse proxy server, and covered installation and basic configuration. In this article, we will continue to look at some of the configuration options.

We will begin by looking at the settings for the second tab, “Settings“. We already covered “Daemon options” in the previous article. Next is “Storage type“. Here, you can select which storage type Varnish uses. By default, the cache is stored in memory (which makes it faster), but you can switch to disk storage instead. You can also specify the cache size in megabytes here (obviously, you do not want to exceed your RAM/storage space).

Next is “Worker thread configuration“. Here you can set the minimum number of Varnish worker threads, the maximum number of Varnish worker threads, and the worker thread timeout. The next subheading is “General VCL Settings“. The first item is the “Client identity method“. This setting is relevant if you have load balancing configured and you want to load balance based on something the client provides: an IP address, a requested URL, or a user agent. The “Don’t cache posts” check box causes Varnish to stop caching POST requests. The “streaming support” check box enables streaming support. “Session Cache” defines Varnish’s behavior with respect to cookies. By default, Varnish does not cache content with cookies in it. This is the recommended behavior, but should you want to cache even when there are cookies involved (and changing the web application is not possible), you can set up a session cache on a per-user basis here (which makes for a fairly low cache hit ratio). It requires your system to not change the cookie on each page hit. “Cache static content” specifies when Varnish should cache static content. If set to “When possible“, Varnish will cache only static content and per user objects when the session cache is set to it. “Always” will cause Varnish to cache all static content; when cookies are present, Varnish will unset it from the object before caching. “Never” will cause Varnish to never cache static content.


The “Fix gzip compression” check box causes Varnish to ignore compression for image files and unknown compression algorithms if checked. The “Be rFC2616 compliant” check box will cause Varnish to ignore requests that do not comply with RFC 2616. “Forward client IP” lets you select how to log the client’s IP address: “set X-Forwarded For“, “append X-Forwarded For“, “set X-Forwarded-Varnish“, and “unset” to leave it unlogged. “Fetch Grace” allows you to set how long Varnish will keep cached objects.

The last section on this page is “Error Settings“. “Retries” determines how many times Varnish will try before it sends an error message. “Saintmode” determines how long Varnish will send cached objects from a backend which has gone offline. Finally, “Custom HTML error message” lets you paste a custom html error page code.

That covers the settings in the Varnish “Settings” tab. In the next article, we will continue our look at Varnish settings.


External Links:

Caching, even when cookies are present at www.varnish-cache.org

Reverse Proxy Services with Varnish (Part One)

reverse proxy

Backend server settings page for Varnish under pfSense 2.1.3.

I recently ran a series of articles on Squid, a proxy server which began life as a client-side cache and can be used under pfSense as such. In contrast, Varnish is a reverse proxy HTTP accelerator designed for content-heavy dynamic web sites. It is focused exclusively on HTTP, unlike other reverse proxy servers that support other network protocols.

The Varnish Reverse Proxy: Architecture and Performance

Varnish works by storing data in virtual memory and leaving the task of deciding what is stored in memory and what gets paged out to disk to the operating system. This helps avoid the situation where the operating system starts caching data while it is moved to disk by the application. In addition, Varnish is heavily threaded, with each client connection being handled by a separate worker thread, and an overflow queue to handle incoming connections once the configured limit on the number of active worker threads is reached. When the overflow queue limit is reached, incoming connections will be rejected.

Varnish uses the Varnish Configuration Language (VCL), which is used to write hooks that are called at critical points in the handling of each request. When a VCL script is loaded, it is translated to C, compiled into a shared object by the system compiler, and loaded directly into the accelerator.


The creators of Varnish claim that it speeds up delivery by a factor of 300-1000 times, depending on your architecture. When the Varnish cache is enabled, performance is bound by the speed of the network, thus turning web server performance into a non-issue. It is free software licensed under a two-clause BSD license, also known as the FreeBSD license.

The Varnish Reverse Proxy: Installation and Configuration

reverse proxy

The Settings tab in Varnish.

Installation of Varnish under pfSense is similar to installation of other packages. From the pfSense web GUI, navigate to System -> Packages and scroll down. You will see both “Varnish” and “Varnish 3” on the packages list. Click on the “plus” sign to the right of the listing for Varnish 3 to install the Varnish reverse proxy. On the next screen, press the “Confirm” button to confirm that you want the package installed. It should take about 3-4 minutes for installation to complete.

Once Varnish is installed, you will have a new item on the “Services” menu called “Varnish“. If you navigate there, you will see seven tabs covering different settings for the Varnish reverse proxy server: “Backends“, “Settings“, “Custom VCL“, “LB Directors“, ‘XMLRPC Sync“, “View Configuration“, and “VarnishSTAT“. Before you can enable Varnish, you need to configure at least one backend server, so click on the “Backends” tab.

Varnish has a concept of “backend” or “origin” servers. A backend server is the server providing the content Varnish will accelerate. To add a backend server, click on the “plus” button on the “Backends” tab. There are four sections on this page. “Backend settings” covers the most basic settings. “Backend name” is the name of the backend web server, and “IP address” is the IP address (on your local network) of the backend web server. At “Port“, you enter the port of the web server (usually 80), and for “Description“, you can enter a description.

The next section is “Performance metrics“. “First byte timeout” represents the time to wait for the first byte for the backend and the time to wait between each received byte. “Connect timeout” is simply the time to wait for a backend connection.

Next is “Probe settings“. If this is configured properly, the Varnish reverse proxy will check the health of each backend with a probe. “Probe URL” is the URL that Varnish will use to ensure that the backend is healthy. “Probe interval” specifies how often we should poll. “Timeout” simply specifies the timeout of the probe; this is the amount of time (in seconds) that Varnish will wait for a backend probe. “Probe Window” specifies how many probes Varnish will retain when considering backend health. “Threshold” specifies how many of the probes specified by “Window” must have succeeded for us to consider the backend healthy. Finally, checking the “Disable Probe” check box disables probing for this backend. In the last section, “Backend Mappings“, you can map either a hostname or a URL to this server. It can either match a string or a regular expression.

The second tab is the “Settings” tab. Under “Daemon options” you can enable Varnish by checking the “Enable Varnish” check box. At “Listening port” you can set the port Varnish will listen on. The “Management interface” specifies the IP address and port for the management interface, and “Advanced startup options” specifies any startup options to include in the rc.d configuration file.

If you specify settings for these options and check the “Enable Varnish” check box, you should have Varnish up and running and working with any web servers you specified as a backend. There are several other settings that you might find useful, and we will cover those in the next article.


External Links:

The official Varnish web site

Varnish at Wikipedia

pfSense Configuration: A Scrounger’s Guide (Part Three)

pfSense configuration

The General Information page in the pfSense setup wizard.

In the last article, we covered selection and installation of hardware for our new pfSense system, as well as the installation and initial pfSense configuration. In this article, we will continue with pfSense configuration.

At this point, I connected the WAN interface to the cable modem. On the LAN side, I decided to connect the LAN interface to an Asus RT-N12 router running in WAP (wireless access point) mode. pfSense would act as a DHCP server and would be responsible for routing, while the RT-N12’s job would be to enable wireless devices to connect to the network.

pfSense Configuration: The Setup Wizard

Now I could complete pfSense configuration by accessing the web configurator via the pfSense box’s LAN IP address (in this case, 192.168.2.1). I typed the address into a web browser tab, specified the default admin password (pfsense), and began. When accessing the web GUI for the first time, you will be presented with a wizard which will guide you through the initial configuration.

The first pfSense configuration screen (after clicking on the first “Next” button) is the “General Information” settings. Here, you can specify a hostname and domain name. You can also specify the primary and secondary DNS servers; you can use your ISP’s DNS servers or other DNS servers. The last setting on this page is the “Allow DNS servers to be overridden by DHCP/PPP on WAN”, which causes pfSense to use the DNS servers specified by your ISP rather than the ones you specify here. You can usually leave this checked, but if you have a setup that requires pfSense to use a specific DNS server, then you want to make sure it is not checked. I could not think of any reason it would be a problem, so I left it checked.


On the next screen, you can enter the time server hostname and the timezone. I left the time server hostname set to the default, and set the timezone to US/Eastern, and clicked on “Next“. The next screen allows you to set the admin password. You will want to change the password from the default, so here we enter a new password (twice).

pfSense configuration

The pfSense dashboard.

That brings us to the end of the setup wizard. Clicking on “Next” will bring up the pfSense dashboard for the first time. As you can see, we now have a system running the current (as of this writing) version of pfSense on an Intel Pentium III. My current network activity doesn’t come close to taxing the system. Moreover, I have plenty of disk space, so if I want to install additional packages like Squid, SquidGuard or nmap, I should be able to do so.

By now, all I needed to do to have a more or less functional pfSense system was to add NAT entries for ports I need to have forwarded. Fortunately, pfSense makes it easy to set up port forwarding (if you leave “Filter rule association” set to the default of “Add associated filter rule”, each NAT entry you add will have a corresponding rule). It took about ten minutes to add my NAT entries (I only had six), and I was done.

In this series, we have demonstrated that the process of repurposing an obsolete computer into an epic pfSense firewall is not that difficult or even time-consuming. Hopefully, if you have been considering undertaking such a project, this will inspire you to do so.

What’s next? First, I want to install some additional packages, such as Squid and SquidGuard. In addition, I still have pfSense on a Neoware thin client, so I am considering setting up a CARP (Common Address Redundancy Protocol) redundancy group in which the Neoware box would act as a backup firewall. This would prevent me from setting up a DMZ on the OPT1 interface, however, since CARP requires a dedicated SYNC interface on each system. This seems like a worthwhile project, however, and if I undertake it, I will be sure to blog about it.


External Links:

Installing pfSense at doc.pfsense.org

pfSense Installation: A Scrounger’s Guide (Part Two)

pfSense installation

The computer I used as my new pfSense box.

In the last article, I discussed my project to turn an old computer into a pfSense firewall and set some guidelines for the project. In this article, I get to configuration of the pfSense box and pfSense installation.

pfSense Installation: Selecting the Hardware

As you recall from part one of this series, the base system requirements for a pfSense installation are:

  • Pentium II or better
  • 256 MB RAM
  • 1 GB of disk space for a standard installation; 512 MB of disk space for embedded systems

I immediately realized the system I used for m0n0wall would not make the grade (too slow and not enough memory). However, I had another old system that might work. I had a Pentium III (733 MHz) with 256 MB RAM. The motherboard for this system died a few months ago; I found a replacement on eBay (for $15), but the system has been running slow ever since. It seemed like an ideal candidate for conversion to a pfSense firewall.


Since I did not want to erase the contents of the original hard drive, I had to find another one to install into the system. I went through a box of old hard drives and found a Western Digital Caviar 22000. With 2 GB of disk space, it had more than enough space for pfSense. I swapped out the original hard drive with the Western Digital.

The next consideration was what network cards to install on the system. You need at least two NICs: one for the WAN and one for the LAN. Installing a third NIC allows you to have an OPT1 interface for a DMZ. Fortunately, there was already one Intel Pro 100 NIC in the computer, and I had a spare two. The Intel Pro 100s are PCI cards, and there are three PCI slots on this motherboard, so I used up all the available PCI slots, but that shouldn’t be a problem. If you need to buy NICs, the folks at pfsense.org recommend purchasing Intel cards (or systems with built-in Intel NICs) up to 1 Gbps. It would behoove you to by Intel PRO 1000s, at least for the LAN and OPT1 interfaces (on the WAN side, using a 100 Mbps NIc will not create a bottleneck for most residential broadband customers). A quick eBay search revealed than PRO 1000s are available for less than $10 (for both PCI and PCI-X interfaces). My Neoware thin client has a 1 Gbps 2-port NIC for the LAN and OPT1 interfaces, and a 100 Mbps NIC for the WAN interface. An upgrade to Intel PRO 1000s on this system is definitely something I will consider in the near future.

pfSense installation

The Compaq Deskpro motherboard recognizes the Samsung drive, so we can proceed.

With the hard drive and NICs installed, I was ready to move the computer over to the test bench and begin pfSense installation. After running setup to make sure the BIOS recognized the Western Digital drive, I put the pfSense CD in and booted the system. When prompted whether to boot pfSense from the CD or run the installer, I hit “I” and invoked the installer. This is where I had my first real setback: although the motherboard’s BIOS recognized the Caviar, pfSense did not, and I therefore could not install pfSense onto it. Fortunately, I had a Samsung sW0434A (total capacity: 4.3 GB) I could install (again courtesy of the box of old hard drives), so I powered down the system and replaced the Western Digital with the Samsung.

pfSense Installation: Options

Once the hard drive had been replaced, I was able to boot pfSense from the CD and begin pfSense installation. When the installer starts, you have a chance to change the video font, change the screenmap, change the keymap, or accept the settings. Since I had no reason to change the defaults, I chose “Accept These Settings“.

On the next screen, you have a choice between quick/easy install and custom install (there are also options to rescue config.xml and reboot). In most cases you can opt for the quick/easy install, but if you do not want to reformat the hard drive, or if you want to partition the hard drive onto which pfSense is installed, or specify a different hard drive geometry than what was detected by pfSense, you want to opt for the custom install. I just wanted to reformat the hard drive and install pfSense onto it, so I opted for “Quick/Easy Install“.

Next, the pfSense installer will give you a choice between installing the standard pfSense kernel, or the embedded kernel (which has no vGA console or keybaord available). I selected “Standard Kernel” and continued. After a few minutes, pfSense was installed, and I was prompted to reboot the system. With pfSense installation complete, I rebooted the system and was ready to run pfSense on this computer for the first time.

When pfSense runs for the first time, it will ask you to assign interfaces. I assigned fxp0 for the WAN and fxp1 for the LAN. [I opted to set up OPT1 from the web configurator, later on]. I also assigned the IP address for the LAN interface.

By now, pfSense installation and configuration was complete, and I had a fully functional pfSense box, but I hadn’t connected it to my network. That’s no fun, so in the next article, I will talk about what happened when I used the new system as my firewall.


External Links:

pfSense Hardware at www.pfsense.org

Intel PRO/1000 GT Desktop Adapter – Overview at www.intel.com

pfSense Hardware: A Scrounger’s Guide (Part One)

pfSense hardware

The Pentium P-233 that served as my m0n0wall firewall/router

When I started using pfSense as my primary firewall, it replaced my previous firewall solution: a Pentium P-233 running m0n0-wall. I eventually switched to a Neoware thin client running pfSense, which I ultimately upgraded to version 2.1.3. The Neoware thin client meets the pfSense hardware requirements for running pfSense on an embedded system, and offered pretty good value for the money – one would be hard-pressed to put together a system more cheaply than these pfSense appliances which has the same features and functionality. Yet while running pfSense from a thin client may be the best option for some users, if you have an old computer that meets the pfSense hardware requirements, this may be the better option. For that reason, I thought it would be an interesting exercise to see how easy (or how hard) it is to turn an old PC into a pfSense firewall.

Indeed, the system I used to run m0n0wall had been scrounged from spare parts. The case and power supply had come from an old barebones system I had bought in the late 1990s. The motherboard/CPU was one of a lot of three I had bought on eBay a few years later, and the CD-ROM was from a group of spare CD-ROM drives I had, as was the floppy drive. I only had 32 MB of RAM initially. I found that with only 32 MB of RAM installed, m0n0wall’s web-based configurator would eventually crash (although the firewall itself would continue to function). I found another 32 MB of RAM on eBay for a few dollars, and my system was complete. The NICs had also been taken from old computers, although I eventually bought a lot of 10 Intel Pro 100 cards for $35. As underpowered as this system might seem, it served ably as my firewall for several years. Thus, I began to wonder if I had any old hardware that could run pfSense, and decided that for my next mini-project, I would take an old computer and turn it into a serviceable pfSense router.


pfSense Hardware: The Guidelines

For this project, I set out some basic guidelines:

  1. The hardware had to meet the general requirements for pfSense hardware. These requirements are listed on the official pfSense web site. For any installation, a Pentium II or better with at least 256 MB of RAM is recommended. For hard drive installations, a 1 GB hard drive is required (and a CD-ROM drive for installation).
  2. When possible, I would scrounge from existing resources to put together a system that would serve as my new pfSense box. If necessary, I would buy new hardware, but only as a last resort.
  3. I was not completely sure what the final system would have installed on it, but I knew at a minimum I wanted to have the most recent pfSense version (2.1.3 at this writing), and probably Squid, SquidGuard, and probably some other packages.
  4. To the fullest extent possible, I would document the process, so I would have a record of what worked (and what didn’t work).

These guidelines should provide a rough road map for this project. In the next article, I will cover the selection of hardware, putting together my pfSense box, and installing pfSense onto it.


External Links:

Hardware for pfSense at pfsense.org – pfSense hardware requirements guide

Web Filtering with SquidGuard: Part Two

web filtering

The General settings tab in SquidGuard in pfSense 2.1.3.

In the previous article, we discussed how to install SquidGuard and began to look at configuration options, focusing on blacklists and access control lists. In this article, we continue our look at SquidGuard configuration.

Filtering Sites By Domain Name, URL, or Regular Expression

We will begin by considering sites that you need to allow your users to access. To prevent these sites from being blocked, you could create a new target category and add a list of domains or URLs that should not be blocked. To do this, click on the “Target categories” tab. From here, click on the plus symbol to add a new category. Each category must be assigned a name (no spaces allowed). The new target category can filter by domain name, URL, or by an expression. Filtering by domain will grant access to the main site and any sub pages on it. Entering a URL will allow access to that exact web page and nothing more. Expressions allow the administrator to grant access based on certain keywords. When you have created all the categories you want to create, press the “Save” button. Then go back to either the “Common ACL” or “Group ACL” tab (wherever you created the rule) and select the option of “Whitelist” for your new category. [You can just as easily select the “Deny” option and blacklist all sites in the category.]


In addition to domain and URL filtering, administrators can create filters using regular expressions in SquidGuard. These types of filters are useful if you want to search for certain strings of text in a URL to decide what rule to apply. We won’t go through all the rules of regular expressions, but I should mention that regular expressions typically consist of a series of characters and metacharacters. The metacharacters have a special meaning unless preceeded by an escape sequence (usually a backslash). Here are some of the more important metacharacters:

  • . : Matches any single character – for example, a.c matches aac, abc, etc. Putting brackets around it causes the dot to be interpreted as a literal dot – [a.c] matches a, ., or c
  • [ ] : A bracket expression; matches a single character or a range contained within the brackets. [abc] matches a, b, or c; [a-z] matches any lowercase letter. – is interpreted literally if it’s the first or last character.
  • [^ ] : Matches any single character that is not contained within the brackets. [^abc] matches any character other than a, b, or c.
  • ^ ; Matches the starting position of any line.
  • * : Mathches the preceding element zero or more times. ab*c matches “ac”, “abc”, “abbbc”, etc.

To create a filter that uses an expression click on the target categories tab and either create a new category or edit an existing one. Enter the expression you want to filter on the expression box, and then press the “Save” button. Then go back to the common or group ACL tab and select Deny, Allow, or Whitelist for your target category.

Here are a few examples of filters in action:

#block some video sites
(.*(metacafe\.|dailymotion\.|/videosearch|video.|myvideo\.|youtube\.com))

#block all .gov sites
(.gov)

#block all .gov and .mil sites
(.gov|.mil)

Squidguard also allows the admin to apply URL filtering based on schdules, which are useful for applying rules at different times during the day, or only on certain days of the week. One way this could be used is for applying strict URL filtering rules during business hours and automatically disable the rules after 5 PM.

To create a time-based rule, click on the “Times” tab. Then click the “plus” sign to create a new schedule. Schedules can be applied using the “Groups” ACL tab. You can create a new group ACL tab (or edit an existing one) and in the “time” dropdown box select the schedule you created. You need to press the “Apply” button on the general tab for the settings to take effect.


External Links:

The official SquidGuard site

URL Filtering – How To Configure SquidGuard in pfSense on hubpages.com

Web Filtering with SquidGuard: Part One

web filtering

The General settings tab for SquidGuard under pfSense 2.1.3.

Now that we’ve covered both Squid and LightSquid, I thought it might be useful to cover some other Squid plugins. In this article, we will cover how to implement web filtering with SquidGuard.

SquidGuard is a URL redirector software, which can be used for web filtering of sites users can access. It uses blacklists to define sites for which access is redirected. Here, we are concerned mainly with SquidGuard installation under pfSense, but it can also be installed under Unix or GNU/Linux. The software’s filtering extends to all computers in an organization, including Windows, Macintosh, UNIX and Linux computers. It was originally developed by Pål Baltzersen and Lars Erik Håland, and was implemented and extended by Lars Erik Håland in the 1990s. The current version is 1.4, released in 2009.

As with other packages, installation of SquidGuard is easy. Just navigate to System -> Packages, scroll down to SquidGuard on the package list, click on the plus button on the right side of the listing, and on the next screen, click the “Confirm” button to confirm installation. The installation will take a few minutes. Once installation is complete, you will have a new menu item under Services called Proxy Filter, with which web filtering can be implemented.


The basic configuration of SquidGuard can be gleaned from the documentation on the official SquidGuard site. The simplest web filtering configuration has a single list of blocked sites and the URL of the page to show when the user tries to access a blocked site. The administrator, though, may choose to define more than one list, each representing a category to block. Finally, sometimes there is a demand to allow specific URLs and domains although they are part of the blacklists for a good reason. In this case, you want to whitelist these domains and URLs. This is generally accomplish in SquidGuard by editing the squidGuard.conf file, but if SquidGuard is installed under pfSense, the basic configuration can be done from the pfSense web GUI.

Web Filtering with SquidGuard: Configuration

You can configure SquidGuard in pfSense by navigating to Services -> Proxy Filter and clicking on the General settings tab. There is a check box at the top; check this box to enable the blacklist. Also on the General settings tab (towards the bottom of the page), you can specify the URL of the blacklist. You can use one of your own blacklists, or you can use one of the publicly available lists on the web. The official SquidGuard site has one at http://www.squidguard.org/blacklists.html. Enter the URL of the blacklist you want to use for web filtering in the appropriate edit box. Once you have done this, press the “Save” button at the bottom of the page.

web filtering

The Blacklist tab in SquidGuard; here you can download blacklists.

Next, click on the blacklist tab and press the “Download” button. Once the download is finished, the status box will display “Blacklist update complete” (it may take several minutes to download).

Once you have uploaded your blacklist, you will need to configure which categories of sites on the blacklist should be either allowed, blocked, or whitelisted. The simplest way to configure it is with the common ACL tab. The common access list settings will apply to all users of the proxy. The next tab is the “Groups ACL” tab, and if you want to apply different rules to ther source networks you should use this tab. This way, you wan have different policies for different networks. When you modify a target rule, you need to click on “Apply” in the General settings tab in order for the changes made to take effect.

In the dropdown box next to each of the target rules, you can select one of the following three web filtering actions:

  • Allow: Grant access to the target category unless blocked by another rule or exception
  • Deny: Block access to sites in the target category
  • Whitelist: Always allow access to the target category (even if blocked by another rule or exception.

At the bottom of the page, there is a check box to block IP addresses in the URL. This will prevent users from bypassing the filter by using the IP of the web site instead of the URL. They can still use the IP addresses of whitelisted sites in the URL, though.

By default, when a user attempts to visit a blocked page, they will see an internal error page indicating that the page was blocked and under which target category it falls. However, you can change the redirect page to a blank page, or any other internal or external URL.

In the next article, we will continue our look at SquidGuard configuration.


External Links:

The official SquidGuard site

URL Filtering – How To Configure SquidGuard in pfSense on hubpages.com

SquidGuard on Wikipedia

Log File Analysis with LightSquid

log file

The “Settings” tab of LightSquid in pfSense 2.1.3.

In the previous article, we covered installation and configuration of Squid under pfSense. In this article, we will cover how to monitor Internet usage with Squid and by installing and using the Squid log file analyzer LightSquid.

LightSquid Installation

Like other packages, LightSquid can easily be installed through the pfSense package manager. To access the package manager, navigate to System -> Packages. Scroll down to LightSquid and click on the plus symbol on the right side of the package to start the installation. Click the “Confirm” button on the next screen to confirm the installation, and LightSquid will be installed within a few minutes. When the installation is complete, there will be a new entry on the “Status” menu called “Proxy Report“.

LightSquid Settings for Log File Analysis

Here are some of the settings available (configurable by navigating to Status -> Proxy report and clicking on the “Settings” tab):

Language: Here you can select the language in which LightSquid reports are displayed in.

Bar color: This setting lets you control the color of the parts in the reports.


Report scheme: Lets you choose the theme for the appearance of the reports. Unless you have a preference for one of the alternate themes, you can leave it as the default of “Base”.

IP resolve method: When parsing the log files, LightSquid attempts to resolve the IP address into domain names. You can change the method it uses to resolve the IPs with this setting. The choices are: IP (just return the IP address); Demo (return AUTHNAME; if AUTHNAME not available return DNSNAME; if DNSNAME not available, return IP address); DNS (return DNSNAME); Simple (return AUTHNAME; if not available return IP); SMB (return smb name of PC); and Squidauth (return AUTHNAME; if not available, return IP address, and allow cryllic names).

log file

A LightSquid user access report.

Refresh sheduler: This setting affects how often the Squid log files are analyzed. Decreasing the value will make the reports stay more up to date but will consume more system resources. Be careful not to set the refresh cycle to occur to frequently. If the system cannot finish one update before another one is requested, you will eventually crash the system.

Skip url: If there are any URLs you do not want to show up in the reports, you can list them here.

To view the reports, click on the “Lightsquid Report” tab. If you get an error message when you click on this tab, you may have to run lightparser.pl (to parse the log files) from the command line on your pfSense box. [To do this, SSH into your pfSense box or access the console directly, drop to the shell, and type in /usr/local/www/lightsquid/lightparser.pl.] Once reports have been generated, you should be able to navigate them. After you select a day you will see a list of clients that accessed the proxy on that day. Once you select a host from this list, you will see a list of all the URLs accessed by that client. Clicking the clock icon at the top of the page will show you the time of the day that each URL was accessed.


External Links:

LightSquid official site

Monitoring Internet Usage with LightSquid and pfSense at hubpages.com

Squid Proxy Configuration in pfSense

Squid proxy

Installing Squid under pfSense 2.1.3.

Squid is a proxy server and web cache daemon. It was originally developed as part of the Harvest project at the University of Colorado Boulder. Further work on the program was completed at the University of California, San Diego (UCSD) and funded via two grants from the National Science Foundation. Duane Wessels forked the last pre-commercial version of Harvest and renamed it Squid, and Squid version 1.0.0 was released in July 1996. It has a number of uses. Under pfSense, it can be used to cache repeated requests.

Squid Proxy Installation

Installing and configuring a Squid proxy server under pfSense is relatively easy. From the pfSense web GUI, go to the top menu and navigate to System -> Preferences. Scroll down to “squid” on the list of packages, and click on the “plus” button on the right to install Squid. On the next screen, press the “Confirm” button to confirm that you want to install Squid. It will take a few minutes for the package installer to unpack and install Squid.

Squid Proxy Configuration

Once installation is complete, “Proxy Server” will show up as an option under Services. Navigate to Services -> Proxy Server to configure Squid. Most users will find the default settings to be acceptable, but there are several settings worth noting. There are 7 tabs in the Squid proxy settings: General, Upstream Proxy, Cache Mgmt, Access Control, Traffic Mgmt, Auth Settings, and Local Users.

Squid proxy

The General settings tab in Squid proxy configuration.

General Settings: This covers the most commonly configured Squid proxy settings. The first setting, “Proxy Interface“, determines which interface or interfaces the proxy will bind to. You probably want to leave this set to “LAN” so that the proxy server is accessible to hosts connected. You probably want to leave “Allow users on interface” checked, to allow users connected to the interface selected in the Proxy Interface field to use the proxy. You probably also want to check the “Transparent proxy” check box so all requests for destination port 80 will be forwarded to the proxy server.

Upstream Proxy: If you want Squid to forward requests to an upstream proxy server, you can enable forwarding here. There are also settings to specify the IP address/hostname of the proxy, TCP and ICP port, and username and password, if the upstream proxy requires them.


Cache Mgmt: This controls a number of settings relating to the cache size. “Hard disk cache size” sets the total amount of hard disk space Squid will use to cache objects. If you have a large hard drive, you can increase this setting to cache more objects; otherwise you can probably leave it at the default value (100 MB).

Memory cache size” is the amount of physical RAM to be used for negative cache and in-transit objects. Objects that squid cannot store in memory end up getting swapped to disk which is much slower than RAM. Squid recommends that this setting should be 50% or less of the installed RAM.

Maximum object size” sets the maximum size of objects saved on disk. The default value is 4 KB. You can increase this parameter to save bandwidth, or lower it to improve speed. Most cache hits tend to take place on small files, although you probably want to increase this parameter from the default of 4 KB.

Access Control: This controls a number of settings regarding who is allowed to access the proxy server. In “Allowed subnets“, you can enter each subnet that is allowed to use the proxy (the proxy interface subnet specified in “General” is already an allowed subnet, so you don’t have to specify it here). “Unrestricted IPs” allows you to specify IP addresses that will not be filtered out by the other access control directives set out in this section. “Banned host addresses” allows you to specify IP addresses that are not allowed to use the proxy. “Whitelist” allows you to specify domains that will be accessible to users that are allowed to use the proxy. “Blacklist” allows you to specify domains that will be blocked to users that are allowed to use the proxy.

Traffic Mgmt: This controls a number of traffic settings. “Maximum download size” limits the maximum total download size to the size specified here (the default is no limit). “Maximum upload size” limits the maximum total upload size to the size specified here (the default is no limit). “Overall bandwidth throttling” specifies the bandwidth throttle for downloads; users will gradually have their download speed increased to this value (default is no throttling). “Per host throttling” specifies the download throttling per host (again, the default is no throttling).

Proxy server

The Authentication settings tab.

Auth Settings: Here, you can enable authentication of Squid proxy users. Squid supports local authentication, as well as authentication through an external LDAP, RADIUS or NT server. The default setting is “None” for no authentication.

Local Users: This allows you to set usernames and passwords for individual users. Assuming authentication is enabled and you chose the local authentication option, users will then be able to use the credentials set here to log in to the Squid proxy server.

For basic Squid proxy usage, the above may be all the information you need. If you really want to understand some of the more advanced options, though, you probably should read the Squid man page. You can use these command-line options by [1] executing a command from the Command prompt option in the Diagnostics menu; [2] SSH-ing into your pfSense box; or [3] accessing pfSense directly from the console.


To shut down Squid, issue this command:

squid -k shutdown

To restart squid, issue this command:

/usr/local/sbin/squid -D

However, it should be noted that pfSense seems to start Squid on its own if it notices Squid is not running. Some of the more interesting options are -u port (to specify a different ICP port than 3130; this can also be done from the pfSense GUI), and -z to create swap directories (useful if you have just deleted the cache and want to recreate the swap directories). There is also a loader.conf.local file in the /boot directory with settings that can be configured. The “Squid Package Tuning” document on doc.pfsense.org suggests changing the kern.ipc.nmbclusters parameter from “0” to “32768”. This increases the amount of memory used for socket buffers, and may improve performance.

External Links:

Squid on Wikipedia

How to Set Up a Transparent Squid Proxy Server Using pfSense at hubpages.com

Squid Package Tuning at doc.pfsense.org

Proxy Servers at pfsensesolution.blogspot.com

© 2013 David Zientara. All rights reserved. Privacy Policy