Traffic Shaping in pfSense: Part Two

Traffic shaping in pfSense

Configuring interfaces in the pfSense traffic shaper wizard.

Wrapping a GUI around the underlying traffic shaping components in pfSense proved to be difficult. Lacking functionality in the underlying system in some areas also limits its capabilities. The traffic shaper was rewritten for pfSense 2.0 and accommodates multiple interfaces.

Traffic to the LAN IP is queued in the same manner as traffic traversing the firewall. If your web interface uses HTTPS, and your traffic shaper queue for HTTPS is filled, it will delay your traffic to the management interface the same as if your HTTPS request were going out to the Internet. If you use pings to the LAN IP from a monitoring system, you may see significant delay for the same reason.


In addition, the shaper is not capable of truly differentiating between protocols. Traffic using TCP port 80 is considered as HTTP, whether it’s really HTTP or it’s P2P application using port 80; traffic using port 443 is considered as HTTPS, and so on. This can be a significant problem in some cases.

Traffic Shaping in pfSense: A Brief Look at PF Rules

Traffic shaping functionality, as with everything else in pfSense, is provided by PF. If you’re willing to write your own rules, this gives you considerable flexibility in configuring traffic shaping. For example, consider the hypothetical from the first article in which there is a backlog of ACK packets on an asymmetric Internet connection. We want to alter the rule set so ACK packets have a higher priority than other packets, so we set up two separate data queues. The result might look something like this:

ext_if="kue0"

altq on $ext_if priq bandwidth 100Kb queue { q_pri, q_def }
    queue q_pri priority 7
    queue q_def priority 1 priq(default)

pass out on $ext_if proto tcp from $ext_if to any flags S/SA \
    keep state queue (q_def, q_pri)

pass in on $ext_if proto tcp from any to $ext_if flags S/SA \
    keep state queue (q_def, q_pri)

Here, a priority-based queue is set up on the external interface ($ext_if) with two subordinate queues. On subqueue has a high priority value of 7 (q_pri), while the other has a low priority value of 1 (q_def). Once a connection is assigned to the main queue, ALTQ inspects each packet’s type of service (ToS) field. ACK packets have the ToS Delay bit set to low, indicating that the sender wanted the speediest delivery possible. When ALTQ sees a low-delay packet and queues of differing priority are available, it will assign the packet to the higher-priority queue.

For those of us who don’t want to be bothered manually rewriting the rules, there’s the traffic shaper wizard. You can access the traffic shaper wizard from the pfSense web interface by navigating to Firewall -> Traffic Shaper and clicking on the Wizard tab. It is generally a good idea to configure traffic for the first time using the wizard. If you need custom rules, you can always step through the wizard, approximate what you need, then make the custom rules afterward. Each screen will setup unique queues, and rules that will control what traffic is assigned into those queues. Should you want to configure everything manually, simply specify your WAN speed at the first screen, then click Next through all the remaining screens without configuring anything.

In the next article, we’ll step though the pfSense traffic shaper wizard.


External Links:

Traffic Shaping Guide at doc.pfsense.org

Traffic Shaping in pfSense: Part One

Traffic Shaping with pfSense

Using the traffic shaping wizard in pfSense 2.2.4.

Traffic shaping, otherwise known as network Quality of Service (QoS), is a means of prioritizing the network traffic crossing your firewall. Without traffic shaping, all packets are processed on a first in/first out basis by your firewall. QoS offers a means of prioritizing different types of traffic, ensuring that high priority services receive the bandwidth they need before lesser piroity services. The traffic shaper wizard in pfSense gives you the ability to quickly configure QoS for common scenarios, and custom rules may also be created for more complex tasks.

Traffic shaping is essentially like a gatekeeper in which important packets are prioritized, while regular packets have to wait, and low-priority packets are kept out until there is not enough higher-priority traffic to use up the bandwidth.

There are traffic shaping queues and traffic shaping rules. The queues are where bandwidth and priorities are actually allocated. Traffic shaping rules control how traffic is assigned into those queues. Rules for the shaper work in a similar way to firewall rules, and allow similar matching characteristics. If a packet matches a shaper rule, it will be assigned into the queues specified by that rule.

The idea of raising or lowering the priority of packets is a simple one, but one which has many possible applications. Here are a few ways in which traffic shaping can be used.

Traffic Shaping in pfSense: Prioritizing ACK Packets

Asymmetric Internet connections (where the download speed differs from the upload speed, usually in such a way that download speed > upload speed) are commonplace, especially with DSL. Some links are so out of balance that the maximum download speed is almost unattainable because it is difficult for the client to send back enough ACK packets to keep traffic flowing. ACK packets are transmitted back to the sender by the receiver to indicate that data has been successfully received, and to signal that it is OK to send more. If the sender does not receive ACKs in a timely manner, TCP’s congestion control will be invoked and it will slow down the connection.

This can happen if you are uploading and downloading simultaneously over an asymmetric connection. The uploading part of the circuit is full from the file upload, and there is little room to send ACK packets which allow downloads to keep flowing. By using the shaper to prioritize ACK packets, you can achieve faster, more stable download speeds on asymmetic links. [This is not as important on symmetric links, but it may still be desirable if the available outgoing bandwidth is heavily utilized.]

Traffic Shaping in pfSense: VoIP, Online Gaming and Peer-to-Peer Traffic

If your VoIP calls use the same circuit as data, then uploads and downloads may degrade your call quality. pfSense can prioritize the call traffic above other protocols and ensure that the calls make it through clearly without breaking up. If there are other transfers occurring simultaneously when the VoIP call is in progress, the speed of the other transfers will be reduced to leave room for the calls.

There are also options in pfSense to give priority to the traffic associated with network gaming. Similar to prioritizing VoIP calls, the effect is that even if you are downloading while playing, the response time of the game should be nearly as fast as if the rest of your connection were idle.

In addition, by lowering the priority of traffic associated with known peer-to-peer ports, you will have the assurance that even if these programs are in use, they won’t hinder other traffic on your network. Due to peer-to-peer traffic’s lower priority, other protocols will be favored over P2P traffic, which will be limited when any other services need the bandwidth.

In the next article, we will discuss some of the limitations of pfSense’s traffic shaper.

External Links:

Traffic Shaping at Wikipedia

pfSense Multi-WAN Configuration: Part Seven

pfSense multi-WAN

Changing the weight of a WAN gateway in pfSense 2.2.4.

There are some scenarios where you may want to only use failover. Some pfSense users have a secondary backup Internet connection with a low bandwidth limit, and only want to use that connection if their primary connection fails, and only while it is down. Failover pools allow you to do this. Another possible usage for failover pools is when you want to make sure a certain protocol or destination always uses only one WAN.

pfSense Multi-WANs: Configuring Weights

In pfSense 2.2, you can configure a weight/preference value to WANs. As described in an earlier article, you can set up a gateway group where the WAN interfaces have different priorities. In a gateway group, interfaces at the same tier have equal priority. Lower-numbered tiers have higher priority than higher-numbered tiers. For example, if we set WAN and WAN1 to Tier 1 and WAN2 to Tier 2, WAN and WAN1 will have equal priority. WAN2 will only come into use if both WAN and WAN1 are down. This is good, but what if WAN is our high-speed connection and WAN1 is a DSL connection, and therefore we want WAN to get the bulk of the traffic?

In this case, we can navigate to System -> Routing. The Gateway tab should be selected by default; if not, click on Gateway. Press the e button next to the entry for WAN. On the next page, press the Advanced button. The first entry in this section should be Weight. Using the dropdown box, change the weight to 2. The weight sets the ratio for use of a gateway. Once we change WAN’s weight to 2, there will be two Tier 1 gateways (WAN and WAN2) with weights of 2 and 1 respectively. Thus, out of 3 connections, 2 will use WAN, and 1 will use WAN1, so WAN should get two-thirds of the traffic. Similarly, we could change WAN’s weight to 3, so that WAN will get three-fourths of the traffic. When we are done changing the weight, we need to press Save at the bottom of the page and then press Apply Changes on the next page.

Note that this distribution is strictly balancing the number of connections. It does not take interface throughput into account. This means your bandwidth usage will not necessarily be distributed equally, though in most environments it works out to be roughly distributed as configured over time. This also means if an interface is loaded to its capacity with a single high throughput connection, additional connections will still be directed to that interface. Ideally you would want to distribute connections based on interface weights and the current throughput of the interface.

External Links:

Network Load Balancing on Wikipedia

pfSense Multi-WAN Configuration: Part Six

pfSense multi-WAN

In the previous articles, we covered the basics of multi-WAN configuration with pfSense. In this article, we will cover how to tailor your configuration to your particular needs.

pfSense Multi-WAN: Bandwidth Aggregation and Service Segregation

One of the main reasons for configuring a multi-WAN setup is bandwidth aggregation. With load balancing, pfSense can help you accomplish this. The caveat, though is that if you have two WAN circuits of X Mbps each, you can’t get 2X of throughput with a single client connection. Each individual connection must be tied to only one specific WAN. This is true of any multi-WAN solution: you cannot simply aggregate the bandwidth of two Internet connections into a single large data pipe without some involvement from the ISP. With load balancing, since individual connections are balanced in a round robin fashion, you can achieve 2X Mbps of throughput using two X Mbps circuits, just not with a single connection. Applications that utilize multiple connections, however, such as many download accelerators, will be able to achieve the combined throughput capacity of two or more connections.

This the real advantage of load balancing: in networks with numerous individual machines accessing the Internet, load balancing should enable you to achieve near the aggregate throughput by balancing the many internal connections out all of the WAN interfaces.

In some situations, you may have a reliable, high-quality Internet connection that has low bandwidth, or high costs for excessive transfers, and another connection that is fast but is of lesser quality. In these situations, it may behoove you to segregate services between the two Internet connections by their priority. High priority services may include VoIP, traffic destined to a specific network such as an outsourced application provider, some specifid protocols used by critical applications, amongst other options. Low priority traffic can be defined as any permitted traffic that does not match the list of high priority traffic. You can set up your policy routing rules in such a way as to direct the high priority traffic (e.g., VOIP traffic) out the high quality Internet connection, and also direct the lower priority traffic out the lesser quality connection.

External Links:

Network Load Balancing on Wikipedia

pfSense Multi-WAN Configuration: Part Five

pfSense multi-WAN

Viewing the load balancer status in pfSense 2.2.4.

Once you have configured your multi-WAN setup, you will want to verify its functionality. In this article, we will cover how to test each component of your multi-WAN setup.

If you have configured failover, you will want to test it after completing your configuration to ensure it functions as you desire, otherwise you might be in for an unpleasant surprise when one of your Internet connections fail. Navigate to Status -> Load Balancer and ensure all your WAN connections show as “Online“ under Status. If they do not, verify your monitoring IP configuration as discussed in previous articles on this site.


pfSense Multi-WAN: Simulating a Failure

There are a number of ways you can simulate a WAN failure, depending on the type of Internet connection being used. In most cases, the easiest way to simulate it is to unplug the target WAN interface’s Ethernet cable from the firewall.

For cable and DSL connections, you will also want to try powering off your modem, and unplugging the coax or phone line from the modem. For T1 and other types of connections with a router outside of pfSense, try unplugging the Internet connection from the router and also turning off the router itself.

All of the abovementioned testing scenarios will likely end with the same result, but there are some circumstances where trying all these things individually will find a fault you might not have otherwise noticed until an actual failure. For example, assume you are using a monitor IP assigned to your DLS or cable modem. Thus when the coax or phone line is disconnected, simulating a provider failure rather than an Ethernet or modem failure, the monitor ping still succeeds since it is pinging the modem. As far as pfSense is concerned, the connection is still up, so it will not fail over even if the connection is actually down. There are other types of failure that can similarly only be detected by testing all the individual cases where failure is possible. After creating a WAN failure, refresh the Status -> Load Balancer screen to check the current status.

The easiest way to verify a HTTP load balancing configuration is to visit one of the websites that displays the public IP address from which you are coming. There is a page on the pfSense website for this purpose, and there are other sites that serve the same function. Search for “what is my IP address” and you will find numerous websites that will show you what public IP address from which the HTTP request is coming.

If you load one of these pages, and refresh your browser a number of times, you should see your IP address changing if your load balancing configuration is correct. Note if you have any other traffic on your network, you probably will not see your IP address change on every page refresh. Refresh the page 20-30 times and you should see the IP change at least a few times. if the IP never changes, try several different sites, and make sure your browser is really requesting the page again,and not returning something from its cache or using a persistent connection to the server. Manually deleting the cache and trying multiple web browsers are good things to try before troubleshooting your load balancer configuration further.

You can use traceroute to test load balancing (or tracert in Windows). Traceroute allows you to see the network path taken to a given destination.

The real time traffic graphs under Status -> Traffic Graph are useful for showing the real time throughput on your WAN interfaces. You can only show one graph at a time per browser window, but you can open additional windows or tabs in your browser and show all your WAN interfaces simultaneously. The Dashboard widget enables the simultaneous display of multiple traffic graphs on a single page. The RRD traffic graphs accessible under Status -> RRD Graphs are useful for longer-term and historical evaluation of your individual WAN utilization.


External Links:

Network Load Balancing on Wikipedia

pfSense Load Balancing

pfSense load balancing

Creating a load balancing pool in pfSense 2.2.4.

In the previous article, we covered how to set up load balancing for a multi-WAN configuration. In this article, we will cover load balancing and failover in cases that don’t involve multiple WAN interfaces.

pfSense Load Balancing

To configure a pfSense load balancing pool, log into the pfSense web GUI and navigate to Services -> Load Balancer. On the Pools tab, click the plus button. This will take you to the Load Balancer configuration page.

In the Name field, fill in a name for the failover pool up to 16 characters in length. This will be the name used to refer to this pool in the Gateway field in the firewall rules. In the Description field, you may enter a description for your own reference. The Description field is optional and does not affect functionality, while the Name field is required. For Mode, select either Load Balance to set up a load balancing pool. In Port, enter the port your servers are listening on, and in Retry specify how many times to check a server before declaring it to be down. In Monitor, select the protocol to be used for monitoring the servers (usually ICMP). In Server IP Address, you enter the IP address that will determine whether the chosen interface is available. If pings to this address fail, this interface is marked as down and is no longer used until it is accessible again.

After selecting an interface and choosing a monitor IP, you can press the Add to pool button to add the interface. After adding the first interface to the pool, select the second interface, sselect its monitor IP, and press Add to pool again. When finished adding interfaces to the pool, press save, and then press Apply changes on the next page.


Failover refers to the ability to use only one WAN connection, but switch to another WAN if the preferred connection fails. This is useful in situations where you want certain traffic, or all of your traffic to utilize one specific WAN connection unless it is unavailable.

To set up a failover group, navigate to Services -> Load Balancer, and click on the plus button, the same as you would when configuring a load balancing pool. In the Name field, fill in a name for the failover pool (again, up to 16 characters in length). In the Description field, you may enter a descripton for your reference.

For the Mode, select Failover. In Port, enter the port your servers are listening on, and in Retry, enter the number of times the server should be checked before being declared to be down. In the Monitor field, set a protocol for monitoring, and in Server IP Address, set the monitor IP. Once you have entered all this information, you can press the Add to pool button. You have added the first interface.

Since this is a failover pool, the first interface aded while be used as long as its monitor IP is responding to pings. If the first interface added to the pool fails, the second interface in the pool will be used. Make sure you add the interfaces to the pool in order of preference. The first in the list will always be used unless it fails, at which point the remaining interfaces in the list are fallen back on in top down order.

Additional interfaces can be added by entering the information for them and clicking Add to pool again. When finished adding interfaces to the pool, press Save, and then Apply Changes on the next page.


External Links:

Inbound Load Balancing on doc.pfsense.org
How to Use pfSense to Load Balance Your Servers on howtoforge.com

pfSense Multi-WAN Configuration: Part Four

pfSense multi-WAN

Setting up multi-WAN load balancing with failover in pfSense 2.2.4

The load balancing functionality in pfSense allows you to distribute traffic over multiple WAN connections in a round-robin fashion. This is done on a per-connection basis. A monitoring IP is configured for each connection, which pfSense will ping, if the pings fail, the interface is marked as down and removed from all pools until the pings succeed again.

pfSense Multi-WAN: Load Balancing 

In pfSense 2.0 and above, Services -> Load Balancer is not used to configure load balancing with a multi-WAN setup. Instead, we use Gateway Groups by navigating to System -> Routing and clicking on the Groups tab. Click the plus button to add a new gateway group.

In the Group Name field, you can enter a group name. The Gateway Priority section is where you configure load balancing. The Tier field determines the link priority in the failover group. Lower-numbered tiers have priority over higher-numbered tiers. Multiple links of the same priority will balance connections until all links at that level are exhausted. If all links in a priority level are exhausted, pfSense will use the next available link in the next priority level.

To illustrate how this works, I created three gateways: WAN, WAN1 and WAN2, as can be seen in the screen capture. Let’s assume that the WAN gateway is my main Internet connection (e.g. a cable modem). Assume that the WAN1 and WAN2 gateways are for my backup Internet connections (e.g. DSL). We want WAN to provide our primary connection to the Internet. When WAN is down, we want our Internet connectivity to be load balanced across WAN1 and WAN2. Therefore, we set WAN to Tier 1 and both WAN1 and WAN2 to Tier 2. Thus, when the higher priority WAN is down, the failover will user WAN1 and WAN2. If either WAN1 or WAN2 go down, pfSense will use the remaining functioning gateway, so that even if two of the gateways are down, we should have some Internet connectivity, albeit with limited bandwidth.

The next field in the table, Virtual IP, allows you to select what virtual IP should be used when the gateway group applies to a local Dynamic DNS, IPsec or OpenVPN endpoint. In my example, since I was not setting up the gateway group to be used in any such scenario, I left this field unchanged.

The next field, Trigger Level, allows you to choose which events trigger exclusion of a gateway. The choices are Member Down, Packet Loss, High Latency, and Packet Loss or High Latency. I chose Packet Loss as the trigger. You can enter a brief Description, and press the Save button. On the next page, you’ll need to press the Apply Changes button.

Next, you need to redirect your firewall traffic to the new gateway. Navigate to Firewall -> Rules, and click on the tab of the interface whose traffic you want to redirect (e.g. LAN). Press the plus button to add a new rule. The default settings can be kept for most settings (Source and Destination should both be set to any). Scroll down to Advanced features, and press the Advanced button in the Gateway section. Select the gateway set up in the previous step in the dropdown box. Enter a brief Description, and press the Save button. On the next page, press the Apply Changes button. If you need to redirect traffic on other interfaces, you will have to set up firewall rules for those interfaces as well.

Finally, you need to navigate to System -> General Setup and make sure you have at least one DNS server for each ISP. This ensures that you still have DNS service if one or more gateways goes down. You may need to set up static routes for your DNS servers; part two of this series went into some detail on how to do this.

Once the gateway groups and firewall rules are configured, your multi-WAN load balancing setup should be complete.

External Links:

Network Load Balancing on Wikipedia

pfSense Multi-WAN Configuration: Part Three

pfSense multi-WAN

Advanced Outbound NAT settings in pfSense 2.2.4.

Some multi-WAN configurations require special workarounds because of limitations in pfSense. This article covers those special cases.

Because of the way pfSense distributes traffic over multiple Internet connections using the same gateway IP, you will need to insert a NAT device between all but one of those connections. This is not an elegant solution, but it is a workable one.

pfSense can only accommodate one PPPoE or PPTP WAN connection. Therefore, OPT WAN interfaces cannot use PPPoE or PPTP WAN types. If you need to use PPPoE or PPTP, the best workaround is to use them on your modem or another firewall. Most DSL modems can handle PPPoE and either directly assign your public IP to pfSense or give it a private IP and provide NAT. Public IP passthrough is possible on many modems and is the preferred means of doing this.

pfSense Multi-WAN: NAT Rules

The default NAT rules generated by pfSense will translate any traffic leaving the WAN or an OPT WAN interface to that interface’s IP address. In a default two interface LAN and WAN configuration, pfSense will NAT all traffic leaving the WAN interface to the WAN IP address. The addition of OPT WAN interfaces extends this to NAT any traffic leaving an OPT WAN interface’s IP address. This is the default behavior and is all handled automatically unless Advanced Outbound NAT is enabled. The policy routing rules direct the traffic to the wAN interface used, and the outbound and 1:1 NAT rules specify how the traffic will be translated. If you require Advanced Outbound NAT with multi-WAN, you will need to configure NAT rules for all your WAN interfaces.

When using port forwarding with a multiple WAN setup, keep in mind that each port forward applies to a single WAN interface. A given port can be opened on multiple WAN interfaces by using multiple port forward entries, one per WAN itnerface. The easiest way to accomplish this is to add the port forward on the first WAN connect, then click the plus button to the right of that entry to add another port forward based on that one. Change the interface to the desired WAN interface, and press the Save button.

1:1 NAT entries are specific to a single WAN interface. Internal systems can be configured with a 1:1 NAT entry on each WAN interface, or a 1:1 entry on one or more WAN interfaces and use the default outbound NAT on others. Where 1:1 entries are configured, they always override any other Outbound NAT configuration for the specific interface where the 1:1 entry is configured.

External Links:

Network Load Balancing on Wikipedia

pfSense Multi-WAN Configuration: Part Two

pfSense multi WAN

Configuring the DNS forwarder in pfSense 2.2.4.

In the first article, we covered some basic considerations with a multi-WAN setup. in this article, we will cover multi-WAN configuration.

First, the WAN interfaces need to be configured. You should set up the primary WAN the same way you would in a single WAN setup. Then for the OPT WAN interfaces, select either DHCP or static, depending on your Internet connection type. For static iP conncections, you will need to fill in the IP address and gateway.

Next, you need to configure pfSense with DNS servers from each WAN connection to ensure it is always able to resolve DNS. This is important, especially if your network uses pfSense’s DNS forwarder for DNS resolution. If you only use one ISP’s DNS servers, an outage of that WAN connection will result in a complete Internet outage regardless of your policy routing configuration.


pfSense uses its routing table to reach the configured DNS servers. This means without any static routes configured, it will use only the primary WAN interface to reach DNS servers. Static routes must be configured for any DNS server on an OPT WAN interface to reach that DNS server. Static routes must be configured for any DNS server on an OPT WAN interface, so pfSense uses the correct WAN interface to reach that DNS server.

This is required for two reasons. [1] Most ISPs prohibit recursive queries from hosts outside their network. Thus, you must use the correct WAN interface to access that ISP’s DNS server. [2] If you lose your primary WAN interface and do not have a static route defined for one of your other DNS servers, you will lose all DNS resultion ability in pfSense, since all DNS servers will be unreachable when the system’s default gateway is unreachable. If you are using pfSense as your DNS server, this will result in a complete failure of DNS for your network.

pfSense Multi-WAN: Static IPs vs. Dynamic IPs

A setup that has all static IPs on the WAN interfaces is easy to handle, as each WAN has a gateway IP that will not change. Dynamic IP WAN interfaces, on the other had, pose difficulties because their gateway is subject to change and static routes in pfSense must point to a static IP address. This usually is not a major problem, since only the IP address changes while the gateway remains the same. If your OPT WAN public IP changes subnets (and therefore gateways) frequently, use of the DNS forwarder in pfSense is not an acceptable solution for redundant DNS servcies; you will still have no reliable means of reaching a DNS server over anything other than the WAN interface.

pfSense multi-WAN

Configuring DNS servers with multiple WAN interfaces in pfSense 2.2.4.

With dynamic IP WANs, you have two alternatives. Because traffic from the inside networks is policy routed by your firewall rules, it is not subject the the limitation of requiring static routes. You can either use DNS servers on the Internet on all your internal systems, or use a DNS server or forwarder on your internal network. As long as DNS requests are initiated from inside your network and not on the firewall itself (as it is in the case of the DNS forwarder), static routes are not required and have no effect on traffic initiated inside your network when using policy routing.

A second option to consider is using one of your DNS server IPs from each Internet connection as the monitor IP for that connection. This will automatically add the appropriate static routes for each DNS server.

If you have a mix of statically and dynamically addressed WAN interfaces, then the primary WAN should be one of your dynamic IP WANs, as static routes are not required for DNS servers on the primary WAN interface.

The image on the right shows separate DNS servers with a multi-WAN setup in pfSense. In System -> General Setup, you can enter the DNS servers, and you can select the gateway used with the selected DNS server in the dropdown box on the right. As you can see, I have selected different WAN interfaces for each of the DNS servers, so the two WAN interfaces (WAN and WAN1) are not dependent on the same DNS server.


 

External Links:

Network Load Balancing on Wikipedia

© 2013 David Zientara. All rights reserved. Privacy Policy