pfSense Multi-WAN Configuration: Part Six

pfSense multi-WAN

In the previous articles, we covered the basics of multi-WAN configuration with pfSense. In this article, we will cover how to tailor your configuration to your particular needs.

pfSense Multi-WAN: Bandwidth Aggregation and Service Segregation

One of the main reasons for configuring a multi-WAN setup is bandwidth aggregation. With load balancing, pfSense can help you accomplish this. The caveat, though is that if you have two WAN circuits of X Mbps each, you can’t get 2X of throughput with a single client connection. Each individual connection must be tied to only one specific WAN. This is true of any multi-WAN solution: you cannot simply aggregate the bandwidth of two Internet connections into a single large data pipe without some involvement from the ISP. With load balancing, since individual connections are balanced in a round robin fashion, you can achieve 2X Mbps of throughput using two X Mbps circuits, just not with a single connection. Applications that utilize multiple connections, however, such as many download accelerators, will be able to achieve the combined throughput capacity of two or more connections.

This the real advantage of load balancing: in networks with numerous individual machines accessing the Internet, load balancing should enable you to achieve near the aggregate throughput by balancing the many internal connections out all of the WAN interfaces.

In some situations, you may have a reliable, high-quality Internet connection that has low bandwidth, or high costs for excessive transfers, and another connection that is fast but is of lesser quality. In these situations, it may behoove you to segregate services between the two Internet connections by their priority. High priority services may include VoIP, traffic destined to a specific network such as an outsourced application provider, some specifid protocols used by critical applications, amongst other options. Low priority traffic can be defined as any permitted traffic that does not match the list of high priority traffic. You can set up your policy routing rules in such a way as to direct the high priority traffic (e.g., VOIP traffic) out the high quality Internet connection, and also direct the lower priority traffic out the lesser quality connection.

External Links:

Network Load Balancing on Wikipedia

pfSense Multi-WAN Configuration: Part Five

pfSense multi-WAN

Viewing the load balancer status in pfSense 2.2.4.

Once you have configured your multi-WAN setup, you will want to verify its functionality. In this article, we will cover how to test each component of your multi-WAN setup.

If you have configured failover, you will want to test it after completing your configuration to ensure it functions as you desire, otherwise you might be in for an unpleasant surprise when one of your Internet connections fail. Navigate to Status -> Load Balancer and ensure all your WAN connections show as “Online“ under Status. If they do not, verify your monitoring IP configuration as discussed in previous articles on this site.


pfSense Multi-WAN: Simulating a Failure

There are a number of ways you can simulate a WAN failure, depending on the type of Internet connection being used. In most cases, the easiest way to simulate it is to unplug the target WAN interface’s Ethernet cable from the firewall.

For cable and DSL connections, you will also want to try powering off your modem, and unplugging the coax or phone line from the modem. For T1 and other types of connections with a router outside of pfSense, try unplugging the Internet connection from the router and also turning off the router itself.

All of the abovementioned testing scenarios will likely end with the same result, but there are some circumstances where trying all these things individually will find a fault you might not have otherwise noticed until an actual failure. For example, assume you are using a monitor IP assigned to your DLS or cable modem. Thus when the coax or phone line is disconnected, simulating a provider failure rather than an Ethernet or modem failure, the monitor ping still succeeds since it is pinging the modem. As far as pfSense is concerned, the connection is still up, so it will not fail over even if the connection is actually down. There are other types of failure that can similarly only be detected by testing all the individual cases where failure is possible. After creating a WAN failure, refresh the Status -> Load Balancer screen to check the current status.

The easiest way to verify a HTTP load balancing configuration is to visit one of the websites that displays the public IP address from which you are coming. There is a page on the pfSense website for this purpose, and there are other sites that serve the same function. Search for “what is my IP address” and you will find numerous websites that will show you what public IP address from which the HTTP request is coming.

If you load one of these pages, and refresh your browser a number of times, you should see your IP address changing if your load balancing configuration is correct. Note if you have any other traffic on your network, you probably will not see your IP address change on every page refresh. Refresh the page 20-30 times and you should see the IP change at least a few times. if the IP never changes, try several different sites, and make sure your browser is really requesting the page again,and not returning something from its cache or using a persistent connection to the server. Manually deleting the cache and trying multiple web browsers are good things to try before troubleshooting your load balancer configuration further.

You can use traceroute to test load balancing (or tracert in Windows). Traceroute allows you to see the network path taken to a given destination.

The real time traffic graphs under Status -> Traffic Graph are useful for showing the real time throughput on your WAN interfaces. You can only show one graph at a time per browser window, but you can open additional windows or tabs in your browser and show all your WAN interfaces simultaneously. The Dashboard widget enables the simultaneous display of multiple traffic graphs on a single page. The RRD traffic graphs accessible under Status -> RRD Graphs are useful for longer-term and historical evaluation of your individual WAN utilization.


External Links:

Network Load Balancing on Wikipedia

pfSense Multi-WAN Configuration: Part Four

pfSense multi-WAN

Setting up multi-WAN load balancing with failover in pfSense 2.2.4

The load balancing functionality in pfSense allows you to distribute traffic over multiple WAN connections in a round-robin fashion. This is done on a per-connection basis. A monitoring IP is configured for each connection, which pfSense will ping, if the pings fail, the interface is marked as down and removed from all pools until the pings succeed again.

pfSense Multi-WAN: Load Balancing 

In pfSense 2.0 and above, Services -> Load Balancer is not used to configure load balancing with a multi-WAN setup. Instead, we use Gateway Groups by navigating to System -> Routing and clicking on the Groups tab. Click the plus button to add a new gateway group.

In the Group Name field, you can enter a group name. The Gateway Priority section is where you configure load balancing. The Tier field determines the link priority in the failover group. Lower-numbered tiers have priority over higher-numbered tiers. Multiple links of the same priority will balance connections until all links at that level are exhausted. If all links in a priority level are exhausted, pfSense will use the next available link in the next priority level.

To illustrate how this works, I created three gateways: WAN, WAN1 and WAN2, as can be seen in the screen capture. Let’s assume that the WAN gateway is my main Internet connection (e.g. a cable modem). Assume that the WAN1 and WAN2 gateways are for my backup Internet connections (e.g. DSL). We want WAN to provide our primary connection to the Internet. When WAN is down, we want our Internet connectivity to be load balanced across WAN1 and WAN2. Therefore, we set WAN to Tier 1 and both WAN1 and WAN2 to Tier 2. Thus, when the higher priority WAN is down, the failover will user WAN1 and WAN2. If either WAN1 or WAN2 go down, pfSense will use the remaining functioning gateway, so that even if two of the gateways are down, we should have some Internet connectivity, albeit with limited bandwidth.

The next field in the table, Virtual IP, allows you to select what virtual IP should be used when the gateway group applies to a local Dynamic DNS, IPsec or OpenVPN endpoint. In my example, since I was not setting up the gateway group to be used in any such scenario, I left this field unchanged.

The next field, Trigger Level, allows you to choose which events trigger exclusion of a gateway. The choices are Member Down, Packet Loss, High Latency, and Packet Loss or High Latency. I chose Packet Loss as the trigger. You can enter a brief Description, and press the Save button. On the next page, you’ll need to press the Apply Changes button.

Next, you need to redirect your firewall traffic to the new gateway. Navigate to Firewall -> Rules, and click on the tab of the interface whose traffic you want to redirect (e.g. LAN). Press the plus button to add a new rule. The default settings can be kept for most settings (Source and Destination should both be set to any). Scroll down to Advanced features, and press the Advanced button in the Gateway section. Select the gateway set up in the previous step in the dropdown box. Enter a brief Description, and press the Save button. On the next page, press the Apply Changes button. If you need to redirect traffic on other interfaces, you will have to set up firewall rules for those interfaces as well.

Finally, you need to navigate to System -> General Setup and make sure you have at least one DNS server for each ISP. This ensures that you still have DNS service if one or more gateways goes down. You may need to set up static routes for your DNS servers; part two of this series went into some detail on how to do this.

Once the gateway groups and firewall rules are configured, your multi-WAN load balancing setup should be complete.

External Links:

Network Load Balancing on Wikipedia

pfSense Multi-WAN Configuration: Part Three

pfSense multi-WAN

Advanced Outbound NAT settings in pfSense 2.2.4.

Some multi-WAN configurations require special workarounds because of limitations in pfSense. This article covers those special cases.

Because of the way pfSense distributes traffic over multiple Internet connections using the same gateway IP, you will need to insert a NAT device between all but one of those connections. This is not an elegant solution, but it is a workable one.

pfSense can only accommodate one PPPoE or PPTP WAN connection. Therefore, OPT WAN interfaces cannot use PPPoE or PPTP WAN types. If you need to use PPPoE or PPTP, the best workaround is to use them on your modem or another firewall. Most DSL modems can handle PPPoE and either directly assign your public IP to pfSense or give it a private IP and provide NAT. Public IP passthrough is possible on many modems and is the preferred means of doing this.

pfSense Multi-WAN: NAT Rules

The default NAT rules generated by pfSense will translate any traffic leaving the WAN or an OPT WAN interface to that interface’s IP address. In a default two interface LAN and WAN configuration, pfSense will NAT all traffic leaving the WAN interface to the WAN IP address. The addition of OPT WAN interfaces extends this to NAT any traffic leaving an OPT WAN interface’s IP address. This is the default behavior and is all handled automatically unless Advanced Outbound NAT is enabled. The policy routing rules direct the traffic to the wAN interface used, and the outbound and 1:1 NAT rules specify how the traffic will be translated. If you require Advanced Outbound NAT with multi-WAN, you will need to configure NAT rules for all your WAN interfaces.

When using port forwarding with a multiple WAN setup, keep in mind that each port forward applies to a single WAN interface. A given port can be opened on multiple WAN interfaces by using multiple port forward entries, one per WAN itnerface. The easiest way to accomplish this is to add the port forward on the first WAN connect, then click the plus button to the right of that entry to add another port forward based on that one. Change the interface to the desired WAN interface, and press the Save button.

1:1 NAT entries are specific to a single WAN interface. Internal systems can be configured with a 1:1 NAT entry on each WAN interface, or a 1:1 entry on one or more WAN interfaces and use the default outbound NAT on others. Where 1:1 entries are configured, they always override any other Outbound NAT configuration for the specific interface where the 1:1 entry is configured.

External Links:

Network Load Balancing on Wikipedia

pfSense Multi-WAN Configuration: Part Two

pfSense multi WAN

Configuring the DNS forwarder in pfSense 2.2.4.

In the first article, we covered some basic considerations with a multi-WAN setup. in this article, we will cover multi-WAN configuration.

First, the WAN interfaces need to be configured. You should set up the primary WAN the same way you would in a single WAN setup. Then for the OPT WAN interfaces, select either DHCP or static, depending on your Internet connection type. For static iP conncections, you will need to fill in the IP address and gateway.

Next, you need to configure pfSense with DNS servers from each WAN connection to ensure it is always able to resolve DNS. This is important, especially if your network uses pfSense’s DNS forwarder for DNS resolution. If you only use one ISP’s DNS servers, an outage of that WAN connection will result in a complete Internet outage regardless of your policy routing configuration.


pfSense uses its routing table to reach the configured DNS servers. This means without any static routes configured, it will use only the primary WAN interface to reach DNS servers. Static routes must be configured for any DNS server on an OPT WAN interface to reach that DNS server. Static routes must be configured for any DNS server on an OPT WAN interface, so pfSense uses the correct WAN interface to reach that DNS server.

This is required for two reasons. [1] Most ISPs prohibit recursive queries from hosts outside their network. Thus, you must use the correct WAN interface to access that ISP’s DNS server. [2] If you lose your primary WAN interface and do not have a static route defined for one of your other DNS servers, you will lose all DNS resultion ability in pfSense, since all DNS servers will be unreachable when the system’s default gateway is unreachable. If you are using pfSense as your DNS server, this will result in a complete failure of DNS for your network.

pfSense Multi-WAN: Static IPs vs. Dynamic IPs

A setup that has all static IPs on the WAN interfaces is easy to handle, as each WAN has a gateway IP that will not change. Dynamic IP WAN interfaces, on the other had, pose difficulties because their gateway is subject to change and static routes in pfSense must point to a static IP address. This usually is not a major problem, since only the IP address changes while the gateway remains the same. If your OPT WAN public IP changes subnets (and therefore gateways) frequently, use of the DNS forwarder in pfSense is not an acceptable solution for redundant DNS servcies; you will still have no reliable means of reaching a DNS server over anything other than the WAN interface.

pfSense multi-WAN

Configuring DNS servers with multiple WAN interfaces in pfSense 2.2.4.

With dynamic IP WANs, you have two alternatives. Because traffic from the inside networks is policy routed by your firewall rules, it is not subject the the limitation of requiring static routes. You can either use DNS servers on the Internet on all your internal systems, or use a DNS server or forwarder on your internal network. As long as DNS requests are initiated from inside your network and not on the firewall itself (as it is in the case of the DNS forwarder), static routes are not required and have no effect on traffic initiated inside your network when using policy routing.

A second option to consider is using one of your DNS server IPs from each Internet connection as the monitor IP for that connection. This will automatically add the appropriate static routes for each DNS server.

If you have a mix of statically and dynamically addressed WAN interfaces, then the primary WAN should be one of your dynamic IP WANs, as static routes are not required for DNS servers on the primary WAN interface.

The image on the right shows separate DNS servers with a multi-WAN setup in pfSense. In System -> General Setup, you can enter the DNS servers, and you can select the gateway used with the selected DNS server in the dropdown box on the right. As you can see, I have selected different WAN interfaces for each of the DNS servers, so the two WAN interfaces (WAN and WAN1) are not dependent on the same DNS server.


 

External Links:

Network Load Balancing on Wikipedia

pfSense Multi-WAN Configuration: Part One

pfSense multi-WANpfSense incorporates the ability to set up multiple WAN interfaces (multi-WAN), which allows you to utilize multiple WAN connections. This in turn enables you to achieve higher uptime and greater throughput capacity (for example, if the user has one 1.5 Mbps connection and a second 2.5 Mbps connection, their total bandwidth using a multi-WAN setup would be 4 Mbps). It has been reported that some pfSense deployments have used as many as 12 WAN connections, and pfSense may scale even higher than that with the right hardware.

Any additional WAN interfaces are referred to as OPT WAN interfaces. References to WAN refer to the primary WAN interfaces, and OPT WAN to any additional WAN interfaces.

There are several factors to consider in a multi-WAN deployment. First, you’re going to want to use different cabling paths, so that multiple Internet connections are not subject to the same cable cut. If you have one connection coming in over a copper pair, you probably want to choose a secondary connection utilizing a different type and path of cabling. IN most cases, you cannot rely upon two or more connections of the same type to provide redundancy. Additional connections from the same provider are typically a solution only for additional bandwidth; the redundancy provided is minimal at best.

Another consideraton is the path from your connection to the Internet. With larger providers, two different types of connections will traverse significantly different networks until reaching core parts of the network. These core network components are generally designed with high redundancy and problems are addressed quickly, as they have widespread effects.

Whether an interface is marked as down or not is determined by the following ping command:

ping -t 5 -oqc 5 -i 0.7 [IP ADDRESS]

In other words, pfSense sends 5 pings (-c 5) to your monitor IP, waiting 0.7 seconds between each ping. it waits up to 5 seconds (-t 5) for a resoibsem and exits successfully if one reply is received (-o). It detects nearly all failures, and is not overly sensitive. Since it is successful with 80 percent packet loss, it is possible your connection could be experiencing so much packet loss that it is unusable but not marked as down. Making the ping settings more strict, however, would result in false posiitives and flapping. Some of the ping options are configurable in pfSense 2.2.4.

In the next article, we’ll cover WAN interface configuration in a multi-WAN setup.


External Links:

Network Load Balancing on Wikipedia

pfSense Load Balancing: Part One

pfSense Load Balancing

Configuring OPT1 as WAN2 so we can set up a gateway group later on.

In computer networking, load balancing is a method for distributing workloads across multiple computers or a computer cluster, network links, CPUs, storage devices, or other resources. When load balancing is employed, we are looking not just to distribute workloads but to optimize resource use, maximize throughput, minimize response time, and avoid overhead. Using multiple components with load balancing instead of a single company can also increase reliability through redundancy. Load balancing has implicit failover capabilities, since load balancing software is capable of detecting when a resource (e.g. network interface, hard drive) is down and excludes it from the group. Load balancing is usually provided by dedicated software or hardware, such as a multilayer switch or a Domain Name System process, or, as we shall soon see, through pfSense. In this article, I will begin our look at pfSense load balancing.


pfSense Load Balancing: Gateway Configuration

As an example, let’s assume we want to set up multiple WAN interfaces and use load balancing on the group. A default WAN gateway was already created when pfSense was set up. In this example, we will use OPT1 as an additional gateway, and then add both the default interface and OPT1 to a newly-created gateway group, which will employ pfSense load balancing to distribute the workload in round-robin fashion.

The first part of our configuration follows the steps outlined in my <a href=”http://pfsensesetup.com/pfsense-gateways/”>article on gateways</a>. In order to set up our second gateway, first browse to System -> Routing. Click on the “Gateway” tab, if it is not already selected. Click on the “plus” button to add a new gateway. At “Interface”, select OPT1 in the drop-down box. At “Name”, type a name, such as “WAN2”. At “Gateway”, type in the IP address of the network interface (in this case, 192.168.3.1). Check “Default Gateway”, and at “Description”, add a description. Then press the “Save” button to save changes, and, if necessary, press the “Apply changes” button on the next screen.


Next, we will make some changes to the WAN interface (the one described as “Interface WAN Dynamic Gateway”). From the Gateways tab, click on the “edit” button. We can leave “Interface and Name” unchanged, but at “Gateway” we will type an IP address (in this case, 192.168.3.11). Click on “Default Gateway” and change the description to something appropriate (e.g. “WAN gateway). Then press the “Save” button to save the changes, and press the “Apply Changes” button if necessary.

Now we have the two interfaces configured correctly. In part two of this series on pfSense load balancing, we will take our newly-configured WAN interfaces and add them to a gateway group, and configure load balancing for the group.

Erratum: The Original Instructions I Posted Contained an Error, and Here’s Why

It occurred to me when composing Part Two of this article that I made a mistake. I set the WAN gateway to 192.168.4.1 originally; however, since WAN2 is on the 192.168.3.0 subnet, and both WAN gateways will likely be connecting to the same network, they should be on the same subnet. Therefore, I amended the instructions for Part One so that WAN is set to 192.168.3.11. I apologize for any confusion I may have caused.

Other Articles in This Series

pfSense Load Balancing: Part Two

pfSense Load Balancing: Part Three (Web Server Failover)

External Links:

Load Balancing at Wikipedia

Setup Incoming pfSense Load Balancing at doc.pfsense.org

Multi-WAN Load Balancing at pfsensesolution.blogspot.com

© 2013 David Zientara. All rights reserved. Privacy Policy