New Python Site Launched

Anyone who has an interest in the Python programming language might want to take a look at my latest site, I only have a few articles posted so far, but I am setting a goal of posting at least two articles a week. As someone whose forte is in C/C++ programming, I’m looking forward to using the site to further explore the world of Python.

HAProxy Load Balancing: Part One


Configuring HAProxy in pfSense 2.1.5.

HAProxy is an application offering high-availability, load balancing and proxying for TCP and HTTP-based applications. It is particularly suited for high traffic web sites, and is used by a number of high-profile websites including GitHub, Stack Overflow, Reddit, Tumblr, and Twitter. Over the years, it has become the de facto standard open source load balancer, is shipped with most mainstream Linux distributions, and is often deployed by default in cloud platforms. It is written in C and has a reputation for being fast, efficient and stable. HAProxy is free and open source software licensed under the GPL.

HAProxy: Installation and Configuration

To install HAProxy in pfSense, navigate to System -> Packages, scroll down to HAProxy on the list of packages, and press the “plus” button on the right. On the next page, press “Confirm” to confirm installation. It should take about three minutes for installation to complete.

Once HAProxy is installed, there will be a new entry on the “Services” menu called “HAProxy“. From there, you can configure settings. There are three tabs: “Settings“, “Listener“, and “Server Pool“. Under “General Settings“, the “Enable HAProxy” check box allows you to enable the load balancer. “Maximum connections” allows you to set the maximum per-process number of concurrent connections to X. Setting this value too high will result in HAProxy not being able to allocate enough memory. “Number of processes to start” indicates the number of HAProxy processes to start. The default is the number of cores/processors installed. “Remote syslog host” allows you to enter an IP address for the syslog host if there is a remote one.

The next setting, the “Syslog facility” dropdown box, allows you to indicate what type of connection HAProxy will make to syslog. The default is “local0“. By default, your syslog configuration probably does not accept socket connections, and doesn’t have a local0 facility, so if you leave it this way, you will have no HAProxy log. If you want it, configure suslog to accept TCP connections by adding -r to syslogd paramters. You can do this by editing the value of SYSLOGD in /etc/default/syslogd. Then follow these steps:

  1. Set up syslog facility local0 and direct it to file /var/log/haproxy.log by adding this line to /etc/syslog.conf:local0* /var/log/haproxy.log
  2. Restart the syslog service by entering the following command:service syslog restart

The next setting, “Syslog level“, allows you to determine what information is logged. “emerg” only logs emergency notifications, “debug” includes debugging information, “warning” includes warnings, and so on. Finally “Carp monitor” allows you to monitor the CARP interface and only run haproxy on the firewall which is the master. [A CARP, or Common Address Redundancy Protocol, firewall setup involves having a group of redundant firewalls. One firewall is designated as the master, and the others are designated as slaves. If the main firewall breaks down or is disconnected from the network, the virtual IP address allocated for the firewall will be taken by one of the firewall slaves and the service availability will not be interrupted.]

In the next article, we will continue our look at HAProxy configuration.

External Links:

The official HAProxy site

HAProxy on Wikipedia

Bandwidth Monitoring with BandwidthD


Configuring BandwithD in pfSense 2.1.5.

BandwidthD tracks usage of TCP/IP subnets and builds HTML files with graphs to display utilization. Charts are built for individual IP addresses, and by default display utilization over 2 day, 8 day, 40 day, and 400 day periods. Furthermore, each IP address’s utilization can be logged at intervals of 3.3 minutes, 10 minutes, 1 hour or 12 hours in CDF format, or to a backend database server. HTTP, TCP, UDP, ICMP, VPN, and P2P traffic are color-coded.

BandwidthD can produce output in two ways. The first is as a standalone application that produces static HTML and PNG output every 200 seconds. The second is as a sensor that transmits its data to a backend database which is then reported on by dynamic php pages. The visual output of both is similar, but the database-driven system allows for searching, filtering, multiple sensors and custom reports. The BandwidthD plugin for pfSense can present the output in both ways.

BandwithD Configuration and Installation

To install BandwidthD in pfSense, navigate to System->Packages, and scroll down to BandwidthD. Press the “plus” button on the right side, and on the next page, press “Confirm” to confirm installation. The package should complete installation within a few minutes.

Once installation is complete, there should be a new item on the “Services” menu called “Bandwidthd”. Once you navigate there, you will see two tabs: “BandwidthD”, which allows you to configure the settings and “Access BandwidthD”, which allows you to view data. The “BandwidthD” tab has several settings. The “Enable bandwidthd” check box simply enables BandwidthD. The “Interface” drop down box allows you to select the interface to which BandwidthD will bind. “Subnet” allows you to specify the subnet (or subnets) on which BandwidthD will report. The subnet for the interface selected in “Interface” is automatically put in the config, so you do not have to specify it here. Subnets are specified in dotted decimal notation, with a slash and the number of bits of the subnet after the subnet (e.g. The next setting is “Skip Intervals”, which sets the number of intervals to skip between graphing. The default is 0. Each interval is 200 seconds (3 minutes 20 seconds). The next setting, the “Graph cutoff”, is how many kilobytes (KB) must be transferred by an IP before it is graphed (default is 1024).


Viewing bandwidth usage in the BandwidthD web GUI.

The “Promiscuous” check box will put the interface in promiscuous mode to see traffic that may not be routing through the host machine. This will only work on a hub, where all packets are sent to all ports; if the interface is connected to a switch, then the interface will only see the traffic on its port. The “output_cdf” check box allows you to log data to cdf files, while “recover_cdf” reads back the cdf files on startup if enabled.

The “output PostgreSQL” check box allows you to log the data to a PostgreSQL database. If you enable this option, you need to specify a hostname, database name, username and password in the next four edit boxes. In the “sensor_id” field you can enter an arbitrary sensor name. In “Filter” you can specify a Libpcap-format filter string to control what bandwidthd sees. The “Draw graphs” check box draws graphs to graph the traffic if enabled. You can disable this if you want CDF or database output. Finally, “Meta Refresh” sets the interval in seconds at which the browser graph display refreshes. The default is 150; specifying 0 disables it.

Clicking on the “Access BandwidthD” tab will open up a separate browser tab showing a table summarizing the types of traffic on the specified interface (FTP, HTTP, P2P, TCP, UDP, and ICMP), as well as graphs for each of the IP addresses on the interface.

External Links:

The official BandwidthD home page


Configuring BandwithD in pfSense 2.1.5.

Data Link Layer Advertising with ladvd


Configuring ladvd under pfSense 2.1.5.

ladvd sends LLDP (Link Layer Discovery Protocol) advertisements on all available interfaces. This makes connected hosts visible on managed switches. By default, it will run as a privilege-separated daemon. In addition to LLDP, ladvd also supports the following protocols:

  • Cisco Discovery Protocol (CDP): This is a proprietary Data Link Layer protocol developed by Cisco Systems. It is used to share information about other directly connected Cisco equipment, such as the operating system version and IP address. It can also be used for On-Demand Routing, which is a method of including routing information in CDP announcements so that dynamic routing protocols do not need to be used in simple networks.
  • Extreme Discovery Protocol (EDP): Another proprietary Data Link Layer protocol; this one was developed by Extreme Networks.
  • Nortel Discovery Protocol (NDP): A proprietary Data Link Layer protocol used for discovery of Nortel networking devices and certain products from Avaya and Ciena. The Device and topology information may be graphically displayed network management software. This protocol had its origin in the SynOptics Network Management Protocol (SONMP) in the 1990s. When SynOptics and Wellfleet Communications merged in 1994, the merged company was named Bay Networks, and SONMP was rebranded as the Bay Network Management Protocol (BNMP) or the Bay Discovery Protocol (BDP). When Bay Networks was itself acquired by Nortel, they renamed it the Nortel Discovery Protocol (NDP).

ladvd Installation and Configuration

To install ladvd under pfSense, navigate to System -> Packages. On the list of packages, scroll down to “ladvd”, and press the “plus” button on the right side. When the install page comes up, press the “Confirm” button to confirm installation. The installation should not take more than 5 minutes.

Once ladvd is installed, there will be a new item on the “Services” menu called “LADVD”, from which you can configure ladvd. There are two tabs on the the configuration page, “General” and “Status”. Clicking on the “General” tab will allow you to configure all the basic settings. The “Enable ladvd” check box allows you to enable or disable ladvd. The “Interfaces” list box allows you to select the interfaces to which ladvd will bind (you can select multiple interfaces). The “Auto-enable protocols” check box allows you to auto-enable protocols based on received packets. The “Silent” check box will cause ladvd to function without transmitting packets. The “Management interfaces” dropdown box allows you to select the management interfaces for this host; IPv4 and IPv6 addresses on this interface are auto-detected. In the “System Location” edit box, you can specify the physical location of the host. Finally, the last four check boxes allow you to enable specific Link Layer protocols: currently, LLDP, CDP, EDP and NDP are supported. Pressing the “ladvd” button at the bottom of the page saves the settings.

The second tab, “Status”, allows you to see information about ladvd devices as well as a detailed decode of Link Layer traffic.

External Links:

The official ladvd site

Ubuntu man page for ladvd

ModSecurity: Part Two


Configuring site proxies in ModSecurity under pfSense 2.1.5.

In the previous article, we covered installation of ModSecurity and began configuration. In this article, we continue our look at configuration.

We had covered the first five settings on the “Proxy Server Settings” tab. The next setting, the “Use mod_mem_cache” checkbox, enables mod_mem_cache, which stores cached documents in memory. In the next edit box, “mod_mem_cache memory usage”, you can set the memory usage in megabytes. The next setting, the “Use mod_disk_cache” checkbox, implements a disk-based storage manager if checked. The memory usage (in KB) can be specified in the next edit box.

The next setting is to limit the number of POSTS accepted from the same IP address. The purpose of this setting is to help prevent the effects of a Slowloris type of attack. The Slowloris tool provides a means of executing a DoS attack on a web server by not sending a complete set of HTTP request headers. The request will omit the final carriage return/line feed (CRLF) that tells the destination web server that the request has completed so the web server dutifully waits for more request data until it reaches its timeout setting. Now Slowloris can keep it in perpetual waiting mode by sending new requests just before the web server’s timeout setting is reached. Obviously, one of the ways of mitigating the effects of such an attack is to limit the number of POSTS allowed from a single IP address, and this is the objective here.

The next edit box configures the maximum request body size ModSecurity will store in memory. Anything over this limit will be rejected with status code 413 Request Entity Too Large. The next edit box sets the maximum request body size ModSecurity will accept for buffering. The default value is 128 KB.

The next check box, “Enable mod_security protection”, enables mod_security protection for all sites being proxied. The dropbox below that configures the audit logging engine. Possible values are: “On” – log all transactions by default; “Off” – do not log transactions by default; “RelevantOnly” – by default, only log transactions that have triggered a warning or error, or have a status code that is considered to be relevant. The last two edit boxes allow you to specify a custom ErrorDocument and to enter custom ModSecurity rules.

ModSecurity: Configuring Site Proxies

The next tab is “Site Proxies”. By clicking on the “plus” sign on the right side of the page, you can add to the list of sites that will use the ModSecurity Apache proxy. Once you do, you can enter the site name, site webmaster e-mail address, and protocol (HTTP or HTTPS). If you specify HTTPS, then you must specify the “Certificate File”, “Certificate Key File”, and “Certificate Chain File” next. If the “Preserve Proxy hostname” check box is checked, it will pass the “Host:” line from the incoming request to the proxied host instead of the backend IP address. At “Primary site hostname”, you can enter the fully qualified domain name (FQDN) for the website. Finally, you can also specify backend web servers for this site, as well as additional site hostnames.

The third tab, “Logs”, allows you to view the Apache ModSecurity proxy server logs. Pressing the “Clear log” button deletes the log.

External Links:

Mitigating Slow HTTP DoS Attacks at – This blog contains a lot of useful information about computer security, especially ModSecurity

Slowloris HTTP DoS – official site for the Slowloris tool

ModSecurity Reference Manual

ModSecurity: Part One


Configuring settings in ModSecurity under pfSense 2.1.5.

ModSecurity is a open source toolkit for real-time web application monitoring, logging, and access control. It supplies an array of request filtering and other security features to the Apache HTTP Server, IIS, and NGINX. Its capabilities, among other things, include the following:

  • ModSecurity gives you access to the HTTP traffic stream, in real-time, along with the ability to inspect it. This allows you to do real-time security monitoring. ModSecurity also enables you to track system elements over time and perform event correlation.
  • ModSecurity allows you to do virtual patching, a concept of vulnerability migration in a separate layer, where you get to fix problems in applications without having to touch the applications themselves. Virtual patching is available to applications that use any communications protocol, but it is particularly useful with HTTP, because the traffic can generally be well understood by an intermediate device.
  • ModSecurity also allows you to do full HTTP traffic logging. Web servers traditionally do very little when it comes to logging for security purposes; they typically log very little by default, and even with some tweaking, in many cases they do not give you everything you need. ModSecurity gives you the ability to log anything you need, including raw transaction data.
  • ModSecurity allows continuous passive security assessment, a variation of real-time monitoring which can detect traces of many abnormalities and security weaknesses before they are exploited.
  • ModSecurity allows for attack surface reduction, in which you selectively narrow down the HTTP features you are willing to accept, and can do so directly, or through collaboration with other Apache modules.

ModSecurity was designed to be flexible, with a powerful rule language which allows you to do exactly what you need it to do. It also does not interact with a transaction unless you tell it, which leaves the decision-making to you.

There are two deployment options for ModSecurity: embedded and reverse proxy. Embedded involves adding ModSecurity to Apache, which is often a good choice for those who already have their architecture laid out and do not want to change it. Reverse proxy involves installing a dedicated Apache reverse proxy with the ModSecurity module enabled. This is a good option for security practitioners who prefer having a separate security layer. In addition, the standalone reverse proxy will have resources dedicated to it, and you will be able to have more complex rules. One of the disadvantages of this approach is that you will have a new point of failure on your network. In this article, we will cover installation and configuration of ModSecurity under pfSense, which is a reverse proxy deployment.

ModSecurity Installation and Configuration

To install ModSecurity, navigate to System -> Packages, and on the “Available Packages” tab, scroll down to “Proxy Server with mod_security“. Press the “plus” button on the right side of the item to install ModSecurity, and on the next page, press “Confirm” to confirm installation. ModSecurity should be installed within a few minutes. To enable ModSecurity, navigate to Status -> Services, find “apache_mod_security” on the list of services, and press the “Start Service” button (it should look like a Play button). Now there should be a new item on the “Services” menu called “Mod_Security+Apache“.

There are three tabs for ModSecurity settings in pfSense: “Proxy Server Settings“, “Site Proxies“, and “Logs“. Under “Proxy Server Settings“, there are several settings relevant to configuring the web server proxy. You can enter the site administrator’s e-mail address at “Global site E-mail administrator“. You can enter the server’s hostname at “Server hostname” (or leave it blank to bind to all IP addresses on the local network). You can specify the port the proxy server will listen on at “Default Bind to port” (leave it blank to bind to 80).

The next setting is “Additional Addresses“. Here, you can specify additional IP addresses/ports to which the proxy can bind. If you need to specify more than one address, press the “plus” button on the right side to add more.

In the next article, we will continue our look at ModSecurity configuration.

External Links:

The official ModSecurity site


September 2014 Amazon Affiliate Purchases

Here are some of the products readers purchased through my Amazon affiliate links during the month of September 2014:

EnGenius Technologies Long-Range Wireless-N Indoor AP/Bridge (ECB300)

Mikrotik RB951-2N Wireless Router 802.11b/g/n

NZXT Technologies Sentry 3 5.4-Inch Touch Screen Fan Controller Cooling AC-SEN-3-B1

Oriental Furniture Modern Furniture, 6-Feet Helsinki Fabric Japanese Privacy Screen Room Divider, 4 Panel Honey

Disney Infinity Power Disc Complete Series 1 Set of 20

Your purchases through this site’s Amazon affiliate links help keep the lights on at And it doesn’t cost you an extra cent: all commissions come from Amazon’s end of the sale.

Greylisting Advantages and Disadvantages

greylistingIn the previous two articles, we covered installation and configuration of spamd, a useful spam-referral daemon. In this article, we will examine some of the advantages and disadvantages of greylisting.

The Greylisting Process

Before we begin, it might be useful to review some of the basic concepts of spam deferral. The process involves, at its most basic level, dividing hosts into three categories: blacklisted hosts, whitelisted hosts, and greylisted hosts. Blacklisted hosts are hosts that are denied access, while whitelisted hosts are granted access. Greylisted hosts, as the name implies, is a kind of cross between blacklisting and whitelisting: such hosts are temporarily blocked (or, in some cases, temporarily allowed) until an additional step is performed.

When spamd receives e-mail from an unknown host, it records three pieces of data (known as a triplet) for each incoming mail message:

  • The IP address of the host
  • The envelope sender address
  • The envelope recipient address (or just the first of them)

With this data, we follow a basic rule: if we have never seen a triplet before, then we refuse delivery and any others that may come within a certain period of time with a temporary failure.

This data s registered on the mail server’s internal database, along with the time stamp of its first appearance. The e-mail message will be dismissed until the configured period of time is elapsed (in the case of spamd, the default time period is 25 minutes). Temporary errors are defined in the Simple Mail Transfer Protocol (SMTP) as 4xx reply codes. Since SMTP is considered an unreliable transport, the possibility of temporary failures is built into the core spec. Therefore, fully capable SMTP implementations are expected to maintain queues for retrying message transmissions in such cases. During the initial testing of greylisting, it was observed that the vast majority of spam appears to be sent from applications designed specifically for spamming. These applications appear to adopt the “fire-and-forget” methodology, and they attempt to send the spam to one or several hosts for a domain, but never attempt a true retry as a real message transfer agent (MTA) would. Conversely, spamd assumes that if a sender has proven itself able to properly retry delivery, then it is a legitimate sender and not a spammer. Thus, such a host will be whitelisted for a longer period of time, so that future delivery attempts will be unimpeded. As an example, spamd can be configured to require a successful delivery attempt against a registered triplet to be no earlier than 25 minutes after registration and not later than 4 hours after it. Repeated delivery attempts before the 25 minute period will be ignored with the same 4xx reply code. After 4 hours the triplet will be expired, so delivery attempts will register anew. When spamd sees an attempt within the 25 minute – 4 hour window, the connecting host will be whitelisted for a certain period of time (default is 36 days).

In addition to being highly effective at blocking spam (over 95% in initial tests), greylisting also ensures that no legitimate mail is ever blocked. In addition, greylisting has been shown to be extremely effective in blocking e-mail-based viruses, as they also do not tend to retry deliveries. And since these viruses are fairly large, bandwidth and processing savings are considerable as compared to the standard method of accepting delivery and local virus scanning.

This temporary rejection can be issued at different stages of the SMTP dialogue, so that the spam deferral daemon can store more or less data about the incoming message. In addition to whitelisting good senders, a greylisting daemon can provide for exceptions. Greylisting can usually be overridden by a fully validated TLS connection with a matching certificate. In addition, because large senders often have a pool of machines that can send and resend e-mail, IP addresses that have the most-significant 24 bits the same are treated as equivalent. In some cases, SPF records are used to determine the sending pool.

Greylisting Disadvantages

Now that we have presented an overview of greylisting and its advantages, it is time to examine some of the disadvantages of this method.

The biggest disadvantage of greylisting is that for unrecognized servers, it destroys the near-instantaneous nature of e-mail most users have come to expect. Mail from unrecognized servers is typically delayed by 15 minutes, and could be delayed for days. A customer of a greylisted ISP cannot always rely on getting e-mail in a predetermined period of time. This disadvantage is mitigated somewhat by the fact that prompt mail delivery is restored once a server has been recognized and is generally maintained automatically so long as users continue to exchange messages. This disadvantage is apparent when a server accessing a greylising mailserver attempts to reset his credentials to a website that uses e-mail confirmation of password resets. In some cases, the delivery delay imposed by the greylister can exceed the expiry time of the password reset token delivered in the e-mail. In such cases, manual intervention may be required to whitelist the host so the e-mail with the reset token can be used before it expires.

Another problem is that some SMTP senders may interpret the temporary rejection as a permanent failure. Old clients conforming only to the obsolete SMTP specification and ignoring its recommendations may give up on delivery after the first failed attempt, as RFC 821 (the old specification) states that clients “should” retry messages rather than using the word “must”. The problem can affect SMTP clients in unexpected ways. Most MTAs will queue and retry messages, but some do not. A possible solution is to configure the application to use a local SMTP server as an outbound queue, instead of attempting direct delivery. The server operator who employs greylisting can whitelist a client which is known to fail on temporary errors.

Some MTAs, upon encountering the temporary failure message from a greylisting server on the first attempt, will send a warning message back to the original sender of the message. The warning message is not a bounce message, but it is often formatted similarly to one and reads like one. This practice often causes the sender to believe the message has not been delivered, although the message will be delivered successfully, albeit at a later time.

Another problem is that the retry might come from a different IP address than the original attempt. When the source of an e-mail is a server farm or goes out through a relay, it is likely that a server other than the original one will make the next attempt. Moreover, their IPs can belong to completely unrelated address blocks (due to network fault tolerance), thereby defying the simple technique of identifying the most significant part of the address. Since the IP addresses will be different, the recipient’s server will fail to recognize that a series of attempts are related, and refuse each of them in turn. This problem can be partially solved by proactively identifying such server farms and making exceptions for hem. Likewise, exceptions have to be configured for multi-homed hosts and hosts using DHCP.

External Links:

The Next Step in the Spam Control War: Greylisting by Evan Harris on

© 2013 David Zientara. All rights reserved. Privacy Policy