HAProxy Load Balancing: Part One

HAProxy

Configuring HAProxy in pfSense 2.1.5.

HAProxy is an application offering high-availability, load balancing and proxying for TCP and HTTP-based applications. It is particularly suited for high traffic web sites, and is used by a number of high-profile websites including GitHub, Stack Overflow, Reddit, Tumblr, and Twitter. Over the years, it has become the de facto standard open source load balancer, is shipped with most mainstream Linux distributions, and is often deployed by default in cloud platforms. It is written in C and has a reputation for being fast, efficient and stable. HAProxy is free and open source software licensed under the GPL.

HAProxy: Installation and Configuration

To install HAProxy in pfSense, navigate to System -> Packages, scroll down to HAProxy on the list of packages, and press the “plus” button on the right. On the next page, press “Confirm” to confirm installation. It should take about three minutes for installation to complete.

Once HAProxy is installed, there will be a new entry on the “Services” menu called “HAProxy“. From there, you can configure settings. There are three tabs: “Settings“, “Listener“, and “Server Pool“. Under “General Settings“, the “Enable HAProxy” check box allows you to enable the load balancer. “Maximum connections” allows you to set the maximum per-process number of concurrent connections to X. Setting this value too high will result in HAProxy not being able to allocate enough memory. “Number of processes to start” indicates the number of HAProxy processes to start. The default is the number of cores/processors installed. “Remote syslog host” allows you to enter an IP address for the syslog host if there is a remote one.


The next setting, the “Syslog facility” dropdown box, allows you to indicate what type of connection HAProxy will make to syslog. The default is “local0“. By default, your syslog configuration probably does not accept socket connections, and doesn’t have a local0 facility, so if you leave it this way, you will have no HAProxy log. If you want it, configure suslog to accept TCP connections by adding -r to syslogd paramters. You can do this by editing the value of SYSLOGD in /etc/default/syslogd. Then follow these steps:

  1. Set up syslog facility local0 and direct it to file /var/log/haproxy.log by adding this line to /etc/syslog.conf:local0* /var/log/haproxy.log
  2. Restart the syslog service by entering the following command:service syslog restart

The next setting, “Syslog level“, allows you to determine what information is logged. “emerg” only logs emergency notifications, “debug” includes debugging information, “warning” includes warnings, and so on. Finally “Carp monitor” allows you to monitor the CARP interface and only run haproxy on the firewall which is the master. [A CARP, or Common Address Redundancy Protocol, firewall setup involves having a group of redundant firewalls. One firewall is designated as the master, and the others are designated as slaves. If the main firewall breaks down or is disconnected from the network, the virtual IP address allocated for the firewall will be taken by one of the firewall slaves and the service availability will not be interrupted.]

In the next article, we will continue our look at HAProxy configuration.


External Links:

The official HAProxy site

HAProxy on Wikipedia

Failover with CARP in PF: Part One

Failover with CARP in PF

FailoverCommon Address Redundancy Protocol (CARP) is a protocol which allows multiple hosts on the same local network to share a set of IP addresses. Its primary purpose is to provide failover redundancy. It was developed as a non-patent-encumbered alternative to Virtual Router Redundancy Protocol (VRRP), which is defined in RFC 2281 and 3768 and was quite far along towards becoming an IETF-sanctioned standard. It is also an alternative to the Hot Standby Router Protocol (HSRP). It manages failover at the intersection of the link layer and IP layer of the OSI model. Each CARP group has a virtual MAC address, and the CARP advertisements are sent out with this as the source address, which helps switches determine which port the virtual MAC address is currently at.

One of the main purposes of CARP is to ensure that the network will keep functioning even when a firewall goes down because of errors or planned maintenance activities such as upgrades (failover). CARP works by allowing a group of hosts on the same network segment to share an IP address. This group of hosts is referred to as a “redundancy group”. The redundancy group is assigned an IP address that is shared among the group members. Within the group, one host is designated as the master and the other as backups. The master host is the one that currently holds the shared IP; it responds to any traffic or ARP requests directed at it. If the master goes down, one of the backups will inherit the IP address. The handover from one CARP host to another may be authenticated, essentially by setting a shared secret, much like a password. The master then sends out CARP advertisement messages via multicast using the CARP protocol (IP Protocol 112) on a regular basis, and the backup hosts listen for this advertisement. If the advertisements stop, the backup hosts will begin advertising. The advertisement frequency is configurable, and the host which advertises the most frequently is the one most likely to become master in the event of a failure.


CARP is rather similar to VRRP, with a few significant differences:

  • The CARP protocol is address family independent, with the OpenBSD implementation supporting both IPv4 and IPv6.
  • CARP has an “arpbalance” feature that allows multiple hosts to share a single IP address simultaneously (in this configuration, there is a virtual MAC address for each host, but only one IP address).
  • CARP uses a cryptographically strong SHA-1 keyed-hash message authentication code (HMAC) to protect each advertisement.

Some synchronization is required, an at least in the case of PF firewalls, pfsync can handle it. If it is done properly, active connections will be handed over without noticeable interruption. The purpose of pfsync in this context is to act as a virtual network interface specially designed to synchronize state information between PF firewalls. In order to ensure that pfsync meets the packet volume and latency requirements, the initial implementation has no built-in authentication. An attacker who has local link layer access to the subnet used for pfsync traffic can add, change, or remove states from the firewalls. Therefore, it is strongly recommended that a dedicated, trusted network be used for pfsync. This can be as simple as a crossover cable between interfaces on two firewalls.


Failover with CARP: Configuration

Here is a simple example CARP configuration:

sysctl -w net.inet.card.allow=1
ifconfig carp1 create
ifconfig carp1 vhid 1 pass password carpdev em0 \
advskew 100 10.0.0.1 netmask 255.255.255.0

This does the following:

  • Enales receipt of CARP packets (the default setting)
  • Creates a carp(4) interface (carp1)
  • Configures carp1 for virtual host #1, enables a password, sets em0 as the interface belonging to the group, and makes the host a backup die to the advskew of 100 (assuming the master has an advskew less than 100). The shared IP assigned to this group is 10.0.0.1/255.255.255.0.

In the next article, I will cover failover CARP configuration in BSD with some real world scenarios.

External Links:

Firewall Redundancy with CARP and pfsync at openbsd.org

Firewall Failover with pfsync and CARP at countersiege.com

Firewall Failover with CARP

pfSense CARP Firewall Failover

Configuring CARP settings on the primary firewall in pfSense 2.0.

In the past few articles, I have explained some of the typical load balancing and failover scenarios for which pfSense can be used. In this article, I will demonstrate how to set up a CARP redundant firewall using pfSense.

The Common Address Redundancy Protocol (CARP) is a protocol which allows multiple hosts on the same network to share a set of IP addresses. Its primary purpose is to provide failover redundancy, although it can also provide load balancing functionality. If there are two or more computers running CARP, if the primary server fails, then one of the other servers will take over, and pfsyncd will be used to synchronize packet filter states.

A group of hosts using CARP is called a “group of redundancy”; this group allocates itself an IP address which is shared or divided among the members of this group. Within this group, a host is designated as a “Master” and the other members are designated as slaves. The main host is the one that takes the IP address; this host answers any traffic or ARP request brought to the attention of this address. Each host has, in addition to the shared IP address, a second unique IP address is required. For example, if you want to have 2 cluster members, you will need 2 IP addresses for the real interfaces and then an IP for each virtual IP address. So in this case it would amount to 3.


One use of CARP is the creation of redundant firewalls. The virtual IP address allotted to the group of redundancy is indicated as the address of the default router on the computers behind the group of firewalls. If the main firewall breaks down or is disconnected from the network, the virtual IP address will be taken by one of the firewall slaves and the firewall will continue to be available. Setting up a redundant CARP firewall requires two separate an identical pfSense machines. We want each machine to have at least 3 interfaces: a WAN interface, LAN interface, and an interface dedicated to the syncing process (pfsync).

Firewall Failover with CARP, Step One:  Interface Settings

First, we need to set up the WAN, LAN and SYNC interfaces on both machines. On the first system, designated as the primary system, the settings are as follows:

  • WAN: 192.168.4.1
  • LAN: 192.168.1.30
  • SYNC: 192.168.5.1

For the backup system, the settings are as follows:

  • WAN: 192.168.4.2
  • LAN: 192.168.1.31
  • SYNC: 192.168.5.2


Firewall Failover with CARP, Step Two:  Adding Rules and Enabling CARP Synchronization

On both machines, we need to add a firewall rule to allow traffic on the SYNC interface. Navigate to Firewall -> Rules and click on the SYNC interface tab. Click the “plus” button to add a new firewall rule. At “Protocol“, set the protocol to “any“. At “Description“, add an appropriate description. Then press the “Save” button to save the changes and press the “Apply changes” button on the next page if necessary.

Next, we need to go to the backup pfSense machine and enable CARP synchronization. Navigate to Firewall -> Virtual IPs and click the “CARP Settings” tab. In the “State Synchronization Settings (pfsync)” section, check the “Synchronize States” check box. At “Synchronize Interface“, choose SYNC as the interface. Then press the “Save” button to save the changes.

Returning to the primary pfSense machine, we also need to enable CARP synchronization. Again we will navigate to Firewall -> Virtual IPs and click the “CARP Settings” tab. We will again click the “Synchronize States” check box and choose SYNC as the “Synchronize Interface“. In addition, we will check the following:

  • Synchronize Rules
  • Synchronize nat
  • Synchronize Virtual IPs

At “Synchronize Config to IP“, enter the IP address of the backup pfSense system. Also set the “Remote System Password” to the password of the backup pfSense system. Then press the “Save” button and save the changes.

Firewall Failover with CARP, Step Three: Adding Virtual IPs

Finally, we must configure a virtual IP address for the WAN and LAN interfaces on the primary pfSense machine. Navigate to Firewall -> Virtual IPs and click on the “Virtual IPs” tab. Click the “plus” button to add a new virtual IP. At “Type“, choose the CARP radio button. At “Interface”, set the interface to LAN. At “IP Address“, set the address as the single WAN address that will be used for all clients regardless of whether the primary or backup firewall is active. At “Virtual IP Password“, set a password. You can leave “VHID Group” set to 1 and “Advertising Frequency” set to 0. At “Description“, add an appropriate description. Press the “Save” button to save the changes and press the “Apply changes” button on the next page to apply the changes if necessary.

In order to create the virtual IP address for the LAN interface, we can repeat the above steps, with the following modifications:

  • At “Interface“, set the interface to LAN
  • At “IP Address“, set the address to the single LAN address that will be used as the default gateway for all clients regardless of whether the primary or backup firewall is active
  • The default “VHID Group” setting will be 2; leave this unchanged

Now that we have added the virtual IP addresses, configuration is done. The two firewalls will constantly sync their rules, NAT, and virtual IP settings so that if the primary dies, the backup will immediately take its place.

External Links:

Configuring pfSense Hardware Redundancy (CARP) at doc.pfsense.org

How to Configure a pfSense 2.0 Cluster Using CARP

pfSense Virtual IP Addresses: Part Two

In the previous article, I covered setting up pfSense virtual IP addresses with Proxy ARP and CARP. In this article, I will cover pfSense virtual IP addreses with IP Alias and Other types.

pfSense Virtual IP Addresses: IP Alias

pfSense virtual IP addreses

Setting up a pfSense virtual IP address with IP Alias in pfSense 2.0.

IP aliasing is the ability to associate more than one IP address to a network interface. With it, one node on a network can have multiple connections to a network, each serving a different purpose. In a sense, it is the reverse of some of the other scenarios envisioned with virtual IP addresses, in which traffic for one IP address can be directed to several different nodes. IP Alias is:

  • New to pfSense 2.0 (and later)
  • Can be used or forwarded by the firewall
  • Allows entire IP addresses to be added to an interface
  • Works on Layer 2 (Data link layer)
  • Can be in a different subnet than the real interface IP
  • Will respond to a ping request if allowed by firewall rules
  • Can be stacked on top of a CARP VIP to bypass VHID limits and lower the amount of CARP heartbeat traffic. Stacked IP Alias VIPs will synchronize via XMLRPC.
  • Can be used with CARP to add additional subnets to CARP, e.g. Add one unique IP Alias from the new subnet to each node, then add CARP VIPs. Must be added to each node individually as these will not synchronize via XMLRPC or else an IP conflict would occur.


To set up a VIP using IP Alias, start at Firewall -> Virtual IPs and once again click on the “plus” button to add a new virtual IP address. Select “IP Alias” as the “Type” with the radio buttons at the top. For “Interface“, select “WAN” (it should be the default). At “IP Addresses“, type an address at “Address” (everything else should be grayed out). At “Description“, add a description if desired. Click on the “Save” button to save the changes, and then on the next screen, click on “Apply changes” if necessary.


pfSense Virtual IP Addresses: Other

“Other” is the only option of the four provided for VIPs in pfSense 2.0 that can be used if routed to the firewall without needing ARP/Layer 2 messages. Its properties are:

  • Can only be forwarded by the firewall
  • Can be in a different subnet than the interface
  • Cannot respond to pings
  • Can be added individually or as a subnet to make a group of VIPs (As of 2.1)
  • Can be used with CARP, e.g. subnet routed to external CARP VIP

Notably, both IP Alias and Other can be used for clustering (master firewall and standby failover firewall).
To add a virtual IP of type “Other”, again navigate to Firewall -> Virtual IPs and click the “plus” button to add a new virtual IP address. At type, choose “Other” with the radio buttons. At “Interface“, select “WAN” (the default). At “IP Addresses“, type an address at “Address” (all other options are grayed out). At “Description“, add a description if desired. Then press “Save” to save the changes and press “Apply changes” if necessary.

As you can see, setting up pfSense virtual IP addresses is almost trivially easy. The more difficult task is deciding which type of VIP is suited for your requirements and choosing accordingly. The official pfSense documentation site has a table which lists some of the features of the different pfSense VIP types, and I am reprinting it here:

VIP Features Table

VIP Features
VIP Type Version NAT Binding ARP/L2 Clustering In Subnet Subnet Mask ICMP Single/Group
CARP 1.x+ Yes Yes Yes Yes Yes Yes Yes Single
Proxy ARP 1.x+ Yes No Yes No No n/a No (1) Either
Other 1.x+ Yes No No Yes (2) No n/a No (1) Either
IP Alias 2.0+ Yes Yes Yes See Notes No No Yes Single

1: ICMP Column represents responses from the firewall itself without NAT. With 1:1 NAT, any VIP will pass ICMP through to the target device. On 2.1+ ICMP can also be used as a protocol in port forward entries.
2: “Other” type VIPs are for routed subnets, and CARP is irrelevant, so they work

External Links:

What are Virtual IP Addresses? at doc.pfsense.org

pfSense Virtual IP Addresses: Part One

pfSense Virtual IP Addresses

Virtual IP address configuration page in pfSense.

A virtual IP address (VIP or VIPA) is an IP address that is not assigned to a specific single server or network interface card (NIC). Rather, it is assigned to multiple applications on a single server, multiple domain names, or multiple servers. Normally, a server IP address depends on the MAC address of the attached NIC, and only one logical IP may be assigned per card. However, VIP addressing enables hosting for several different applications and virtual appliances on a server with only one logical IP address. VIPs have several variations and implementations, including Common Address Redundancy Protocol (CARP) and Proxy Address Resolution Protocol (Proxy ARP).

pfSense Virtual IP Addresses: Proxy ARP

pfSense allows four types of virtual IP addresses: Proxy ARP, CARP, Other, and IP Alias. In this article, I will cover how to configure pfSense virtual IP addresses using Proxy ARP and CARP.


The different types of virtual IP addresses have slightly varied properties. With proxy ARP, the properties are:

  • Can only be forwarded by the firewall (cannot be used by the firewall)
  • Uses Layer 2 (the data link layer) traffic
  • Can be in a different subnet than the interface
  • Cannot respond to pings
pfSense Virtual IP Addresses

Once the Virtual IP has been entered and saved, it is added to the list.

To configure a Proxy ARP virtual IP address, browse to Firewall -> Virtual IPs and Click the “plus” button to add a new virtual IP address. At type, there are four radio buttons; select the radio button for “Proxy ARP” (it should be the default selection). For “Interface”, select “WAN”. At “IP Address(es)“, select “Single address” for “Type” (this should be the default). At “Address“, specify an IP address. At “Description“, enter a description if desired. Then press “Save” to save the changes and “Apply changes” to apply changes if necessary.

Now, the newly-created VIP should be listed at the “Virtual IPs” tab at Firewall -> Virtual IPs.

pfSense Virtual IP Addresses: CARP

You can also configure a virtual IP with CARP in pfSense 2.0. The properties for a CARP VIP include:

  • Can be used or forwarded by the firewall
  • Uses Layer 2 (data link layer) traffic
  • Should be used in firewall fail-over or load-balancing scenarios
  • Must be in the same subnet as the interface
  • Will respond to pings if configured properly

To set up a CARP virtual IP address, browse to Firewall -> Virtual IPs and click the “plus” button to add a new virtual IP address. At “Type“, select the “CARP” radio button, and at “Interface“, select “WAN” (it should be the default). At “IP address(es)“, specify an IP address. At “Virtual IP Password“, specify a password. At “VHID Group“, choose a group. At “Advertising Frequency“, select a frequency (0 for master). At “Description“, add a description if desired. Then press “Save” to save the changes and “Apply changes” to apply the changes if necessary.

In part two of this series, I will cover setting up virtual IP addresses with IP Alias and Other types.

Once again, the “Virtual IPs” tab under Firewall -> Virtual IPs should display the newly-created VIP within the list of pfSense virtual IP addresses. In part two, I will cover IP aliases (new to pfSense 2.0) and other VIPs.


External Links:

What are Virtual IP Addresses? at doc.pfsense.org

© 2013 David Zientara. All rights reserved. Privacy Policy