Log File Analysis with LightSquid

log file

The “Settings” tab of LightSquid in pfSense 2.1.3.

In the previous article, we covered installation and configuration of Squid under pfSense. In this article, we will cover how to monitor Internet usage with Squid and by installing and using the Squid log file analyzer LightSquid.

LightSquid Installation

Like other packages, LightSquid can easily be installed through the pfSense package manager. To access the package manager, navigate to System -> Packages. Scroll down to LightSquid and click on the plus symbol on the right side of the package to start the installation. Click the “Confirm” button on the next screen to confirm the installation, and LightSquid will be installed within a few minutes. When the installation is complete, there will be a new entry on the “Status” menu called “Proxy Report“.

LightSquid Settings for Log File Analysis

Here are some of the settings available (configurable by navigating to Status -> Proxy report and clicking on the “Settings” tab):

Language: Here you can select the language in which LightSquid reports are displayed in.

Bar color: This setting lets you control the color of the parts in the reports.

Report scheme: Lets you choose the theme for the appearance of the reports. Unless you have a preference for one of the alternate themes, you can leave it as the default of “Base”.

IP resolve method: When parsing the log files, LightSquid attempts to resolve the IP address into domain names. You can change the method it uses to resolve the IPs with this setting. The choices are: IP (just return the IP address); Demo (return AUTHNAME; if AUTHNAME not available return DNSNAME; if DNSNAME not available, return IP address); DNS (return DNSNAME); Simple (return AUTHNAME; if not available return IP); SMB (return smb name of PC); and Squidauth (return AUTHNAME; if not available, return IP address, and allow cryllic names).

log file

A LightSquid user access report.

Refresh sheduler: This setting affects how often the Squid log files are analyzed. Decreasing the value will make the reports stay more up to date but will consume more system resources. Be careful not to set the refresh cycle to occur to frequently. If the system cannot finish one update before another one is requested, you will eventually crash the system.

Skip url: If there are any URLs you do not want to show up in the reports, you can list them here.

To view the reports, click on the “Lightsquid Report” tab. If you get an error message when you click on this tab, you may have to run lightparser.pl (to parse the log files) from the command line on your pfSense box. [To do this, SSH into your pfSense box or access the console directly, drop to the shell, and type in /usr/local/www/lightsquid/lightparser.pl.] Once reports have been generated, you should be able to navigate them. After you select a day you will see a list of clients that accessed the proxy on that day. Once you select a host from this list, you will see a list of all the URLs accessed by that client. Clicking the clock icon at the top of the page will show you the time of the day that each URL was accessed.

External Links:

LightSquid official site

Monitoring Internet Usage with LightSquid and pfSense at hubpages.com

Squid Proxy Configuration in pfSense

Squid proxy

Installing Squid under pfSense 2.1.3.

Squid is a proxy server and web cache daemon. It was originally developed as part of the Harvest project at the University of Colorado Boulder. Further work on the program was completed at the University of California, San Diego (UCSD) and funded via two grants from the National Science Foundation. Duane Wessels forked the last pre-commercial version of Harvest and renamed it Squid, and Squid version 1.0.0 was released in July 1996. It has a number of uses. Under pfSense, it can be used to cache repeated requests.

Squid Proxy Installation

Installing and configuring a Squid proxy server under pfSense is relatively easy. From the pfSense web GUI, go to the top menu and navigate to System -> Preferences. Scroll down to “squid” on the list of packages, and click on the “plus” button on the right to install Squid. On the next screen, press the “Confirm” button to confirm that you want to install Squid. It will take a few minutes for the package installer to unpack and install Squid.

Squid Proxy Configuration

Once installation is complete, “Proxy Server” will show up as an option under Services. Navigate to Services -> Proxy Server to configure Squid. Most users will find the default settings to be acceptable, but there are several settings worth noting. There are 7 tabs in the Squid proxy settings: General, Upstream Proxy, Cache Mgmt, Access Control, Traffic Mgmt, Auth Settings, and Local Users.

Squid proxy

The General settings tab in Squid proxy configuration.

General Settings: This covers the most commonly configured Squid proxy settings. The first setting, “Proxy Interface“, determines which interface or interfaces the proxy will bind to. You probably want to leave this set to “LAN” so that the proxy server is accessible to hosts connected. You probably want to leave “Allow users on interface” checked, to allow users connected to the interface selected in the Proxy Interface field to use the proxy. You probably also want to check the “Transparent proxy” check box so all requests for destination port 80 will be forwarded to the proxy server.

Upstream Proxy: If you want Squid to forward requests to an upstream proxy server, you can enable forwarding here. There are also settings to specify the IP address/hostname of the proxy, TCP and ICP port, and username and password, if the upstream proxy requires them.

Cache Mgmt: This controls a number of settings relating to the cache size. “Hard disk cache size” sets the total amount of hard disk space Squid will use to cache objects. If you have a large hard drive, you can increase this setting to cache more objects; otherwise you can probably leave it at the default value (100 MB).

Memory cache size” is the amount of physical RAM to be used for negative cache and in-transit objects. Objects that squid cannot store in memory end up getting swapped to disk which is much slower than RAM. Squid recommends that this setting should be 50% or less of the installed RAM.

Maximum object size” sets the maximum size of objects saved on disk. The default value is 4 KB. You can increase this parameter to save bandwidth, or lower it to improve speed. Most cache hits tend to take place on small files, although you probably want to increase this parameter from the default of 4 KB.

Access Control: This controls a number of settings regarding who is allowed to access the proxy server. In “Allowed subnets“, you can enter each subnet that is allowed to use the proxy (the proxy interface subnet specified in “General” is already an allowed subnet, so you don’t have to specify it here). “Unrestricted IPs” allows you to specify IP addresses that will not be filtered out by the other access control directives set out in this section. “Banned host addresses” allows you to specify IP addresses that are not allowed to use the proxy. “Whitelist” allows you to specify domains that will be accessible to users that are allowed to use the proxy. “Blacklist” allows you to specify domains that will be blocked to users that are allowed to use the proxy.

Traffic Mgmt: This controls a number of traffic settings. “Maximum download size” limits the maximum total download size to the size specified here (the default is no limit). “Maximum upload size” limits the maximum total upload size to the size specified here (the default is no limit). “Overall bandwidth throttling” specifies the bandwidth throttle for downloads; users will gradually have their download speed increased to this value (default is no throttling). “Per host throttling” specifies the download throttling per host (again, the default is no throttling).

Proxy server

The Authentication settings tab.

Auth Settings: Here, you can enable authentication of Squid proxy users. Squid supports local authentication, as well as authentication through an external LDAP, RADIUS or NT server. The default setting is “None” for no authentication.

Local Users: This allows you to set usernames and passwords for individual users. Assuming authentication is enabled and you chose the local authentication option, users will then be able to use the credentials set here to log in to the Squid proxy server.

For basic Squid proxy usage, the above may be all the information you need. If you really want to understand some of the more advanced options, though, you probably should read the Squid man page. You can use these command-line options by [1] executing a command from the Command prompt option in the Diagnostics menu; [2] SSH-ing into your pfSense box; or [3] accessing pfSense directly from the console.

To shut down Squid, issue this command:

squid -k shutdown

To restart squid, issue this command:

/usr/local/sbin/squid -D

However, it should be noted that pfSense seems to start Squid on its own if it notices Squid is not running. Some of the more interesting options are -u port (to specify a different ICP port than 3130; this can also be done from the pfSense GUI), and -z to create swap directories (useful if you have just deleted the cache and want to recreate the swap directories). There is also a loader.conf.local file in the /boot directory with settings that can be configured. The “Squid Package Tuning” document on doc.pfsense.org suggests changing the kern.ipc.nmbclusters parameter from “0” to “32768”. This increases the amount of memory used for socket buffers, and may improve performance.

External Links:

Squid on Wikipedia

How to Set Up a Transparent Squid Proxy Server Using pfSense at hubpages.com

Squid Package Tuning at doc.pfsense.org

Proxy Servers at pfsensesolution.blogspot.com

© 2013 David Zientara. All rights reserved. Privacy Policy