Shorewall and Multiple Internet Connections

Tom Eastep

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover, and with no Back-Cover Texts. A copy of the license is included in the section entitled “GNU Free Documentation License”.

2009/06/23


Table of Contents

Multiple Internet Connection Support
Overview
/etc/shorewall/providers File
What an entry in the Providers File Does
/etc/shorewall/masq and Multi-ISP
Martians
Example
Routing a Particular Application Through a Specific Interface
Port Forwarding
More than 2 Providers
Applications running on the Firewall
IPSEC
/etc/shorewall/route_rules
Routing Rules
Columns in the route_rules file
Multi-ISP and VPN
Examples
USE_DEFAULT_RT
Gateway Monitoring and Failover
SWPING
Link Status Monitor (LSM)
Two Providers Sharing an Interface
A Complete Working Example

Warning

This document describes the Multi-ISP facility in Shorewall 4.0 and later. If you are running an earlier release, please see the documentation for that release. We strongly recommend that you run Shorewall-perl 4.2 or later if you are going to use this feature.

Warning

Reading just Shorewall documentation is probably not going to give you enough background to use this material. Shorewall may make iptables easy but the Shorewall team doesn't have the resources to be able to spoon-feed Linux policy routing to you (please remember that the user's manual for a tractor doesn't teach you to grow corn either). You will likely need to refer to the following additional information:

Multiple Internet Connection Support

Beginning with Shorewall 2.3.2, limited support is included for multiple Internet connections. Limitations of this support are as follows:

  • It utilizes static routing configuration. As such, there is no provision for reacting to the failure of any of the uplinks.

  • The routing changes are made and the route cache is purged when Shorewall is started and when Shorewall is restarted (unless you specify the "-n" option to shorewall restart). Ideally, restarting the packet filter should have no effect on routing.

  • Prior to Shorewall 3.4.0, the routes and route rules added by this support were not completely removed during shorewall stop, shorewall clear or shorewall restart.

  • For most routing applications, Quagga is a better solution although it requires that your ISPs offer routing protocol support.

Overview

Let's assume that a firewall is connected via two separate Ethernet interfaces to two different ISPs as in the following diagram.

  • eth0 connects to ISP1. The IP address of eth0 is 206.124.146.176 and the ISP's gateway router has IP address 206.124.146.254.

  • eth1 connects to ISP 2. The IP address of eth1 is 130.252.99.27 and the ISP's gateway router has IP address 130.252.99.254.

  • eth2 connects to the local LAN. Its IP configuration is not relevant to this discussion.

Each of these providers is described in an entry in the file /etc/shorewall/providers.

Entries in /etc/shorewall/providers can specify that outgoing connections are to be load-balanced between the two ISPs. Entries in /etc/shorewall/tcrules can be used to direct particular outgoing connections to one ISP or the other. Use of /etc/shorewall/tcrules is not required for /etc/shorewall/providers to work, but you must select a unique MARK value for each provider so Shorewall can set up the correct marking rules for you.

When you use the track option in /etc/shorewall/providers, connections from the Internet are automatically routed back out of the correct interface and through the correct ISP gateway. This works whether the connection is handled by the firewall itself or if it is routed or port-forwarded to a system behind the firewall.

Shorewall will set up the routing and will update the /etc/iproute2/rt_tables to include the table names and numbers of the tables that it adds.

Caution

This feature uses packet marking to control the routing. As a consequence, there are some restrictions concerning entries in /etc/shorewall/tcrules:

  • Packet marking for traffic control purposes may not be done in the PREROUTING table for connections involving providers with 'track' specified (see below).

  • You may not use the SAVE or RESTORE options.

  • You may not use connection marking.

The /etc/shorewall/providers file can also be used in other routing scenarios. See the Squid documentation for an example.

/etc/shorewall/providers File

Entries in this file have the following columns. As in all Shorewall configuration files, enter "-" in a column if you don't want to enter any value.

NAME

The provider name. Must begin with a letter and consist of letters and digits. The provider name becomes the name of the generated routing table for this provider.

NUMBER

A number between 1 and 252. This becomes the routing table number for the generated table for this provider.

MARK

A mark value used in your /etc/shorewall/tcrules file to direct packets to this provider. Shorewall will also mark connections that have seen input from this provider with this value and will restore the packet mark in the PREROUTING CHAIN. Mark values must be in the range 1-255.

Beginning with Shorewall version 3.2.0 Beta 6, you may use may set HIGH_ROUTE_MARKS=Yes in /etc/shorewall/shorewall.conf. This allows you to:

  • Use connection marks for traffic shaping, provided that you assign those marks in the FORWARD chain.

  • Use mark values > 255 for provider marks in this column. These mark values must be a multiple of 256 in the range 256-65280 (hex equivalent 0x100 - 0xFF00 with the low-order 8 bits being zero).

DUPLICATE

Gives the name or number of a routing table to duplicate. May be 'main' or the name or number of a previously declared provider. For most applications, you want to specify 'main' here. This field should be be specified as '-' when USE_DEFAULT_RT=Yes in shorewall.conf

INTERFACE

The name of the interface to the provider. Where multiple providers share the same interface (which is not recommended), you must follow the name of the interface by a colon (":") and the IP address assigned by this provider (e.g., eth0:206.124.146.176). See below for additional considerations.

GATEWAY

The IP address of the provider's Gateway router.

You can enter detect here and Shorewall will attempt to automatically determine the gateway IP address.

Hint: "detect" is appropriate for use in cases where the interface named in the INTERFACE column is dynamically configured via DHCP etc.

The GATEWAY may be omitted (enter '-') for point-to-point links.

OPTIONS

A comma-separated list from the following:

track

If specified, connections FROM this interface are to be tracked so that responses may be routed back out this same interface.

You want to specify 'track' if Internet hosts will be connecting to local servers through this provider. Any time that you specify 'track', you will normally want to also specify 'balance' (see below).

Use of this feature requires that your kernel and iptables include CONNMARK target and connmark match support (Warning: Until recently, standard Debian™ and Ubuntu™ kernels were lacking that support. Both Lenny and Jaunty do have the proper support).

Warning

A bug in Shorewall versions 3.2.0-3.2.10, 3.4.0-3.4.6 and 4.0.0-4.0.2 prevents proper handling of PREROUTING marks when HIGH_ROUTE_MARKS=No and the track option is specified. Patches are available to correct this problem:

Shorewall version 3.2.0-3.4.3: http://www1.shorewall.net/pub/shorewall/3.2/shorewall-3.2.10/errata/patches/Shorewall/patch-3.2.10-2.diff

Shorewall version 3.4.4-3.4.6: http://www1.shorewall.net/pub/shorewall/3.4/shorewall-3.4.6/errata/patches/Shorewall/patch-3.4.6-1.diff

Shorewall-shell version 4.0.0-4.0.2: http://www1.shorewall.net/pub/shorewall/4.0/shorewall-4.0.2/errata/patches/Shorewall-shell/patch-shell-4.0.2-2.diff

Important

If you are using /etc/shorewall/providers because you have multiple Internet connections, we recommend that you specify 'track' even if you don't need it. It helps maintain long-term connections in which there are significant periods with no traffic.

balance

The providers that have 'balance' specified will get outbound traffic load-balanced among them. Balancing will not be perfect, as it is route based, and routes are cached. This means that routes to often-used sites will always be over the same provider.

By default, each provider is given the same weight (1) . You can change the weight of a given provider by following balance with "=" and the desired weight (e.g., balance=2). The weights reflect the relative bandwidth of the providers connections and should be small numbers since the kernel actually creates additional default routes for each weight increment.

Important

If you are using /etc/shorewall/providers because you have multiple Internet connections, we recommend that you specify 'balance' even if you don't need it. You can still use entries in /etc/shorewall/tcrules to force all traffic to one provider or another.

Note

If you don't heed this advice then please read and follow the advice in FAQ 57 and FAQ 58.

Important

If you specify 'balance' and still find that all traffic is going out through only one provider, you may need to install a kernel built with CONFIG_IP_ROUTE_MULTIPATH_CACHED=n. Several users have reported that this change has corrected similar problems.

The SuSE 10.0 kernel is subject to this problem, and a kernel oops may result in this circumstance. SUSE 10.1 and SLES 10 have CONFIG_IP_ROUTE_MULTIPATH_CACHED=n set by default. The source of the problem seems to be an incompatibility between the LARTC patches and CONFIG_IP_ROUTE_MULTIPATH_CACHED.

loose

Do not include routing rules that force traffic whose source IP is an address of the INTERFACE to be routed to this provider. Useful for defining providers that are to be used only when the appropriate packet mark is applied. Should not be specified together with balance.

optional (added in Shorewall 3.2.2)

Shorewall will determine of this interface is up and has a configured IPv4 address. If it is not, a warning is issued and this provider is not configured.

Note

'optional' is designed to detect interface states that will cause shorewall start or shorewall restart to fail; just because an interface is in a state that Shorewall can [re]start without error doesn't mean that traffic can actually be sent through the interface.

Beginning with Shorewall-perl 4.0.3, you can supply an 'isusable' extension script to extend Shorewall's interface state detection.

src=source-address (Added in Shorewall-perl 4.1.5)

Specifies the source address to use when routing to this provider and none is known (the local client has bound to the 0 address). May not be specified when an address is given in the INTERFACE column. If this option is not used, Shorewall substitutes the primary IP address on the interface named in the INTERFACE column.

mtu=number (Added in Shorewall-perl 4.1.5)

Specifies the MTU when forwarding through this provider. If not given, the MTU of the interface named in the INTERFACE column is assumed.

fallback[=weight] (Added in Shorewall-perl 4.2.5)

Indicates that a default route through the provider should be added to the default routing table (table 253). If a weight is given, a balanced route is added with the weight of this provider equal to the specified weight. If the option is given without a weight, an separate default route is added through the provider's gateway; the route has a metric equal to the provider's NUMBER. The option is ignored with a warning message if USE_DEFAULT_RT=Yes in shorewall.conf.

For those of you who are terminally confused between track and balance:

  • track governs incoming connections.

  • balance governs outgoing connections.

COPY

A comma-separated list if interface names. Wildcards specified using an asterisk ("*") are permitted (e.g., tun* ).

When you specify an existing table in the DUPLICATE column, Shorewall copies all routes through the interface specified in the INTERFACE column plus the interfaces listed in this column. Normally, you will list all interfaces on your firewall in this column except those Internet interfaces specified in the INTERFACE column of entries in this file.

What an entry in the Providers File Does

Adding another entry in the providers file simply creates an alternate routing table for you. The table will usually contain two routes:

  1. A host route to the specified GATEWAY through the specified INTERFACE.

  2. A default route through the GATEWAY.

Note that the first route is omitted if "-" is specified as the GATEWAY; in that case, the default route does not specify a gateway (point-to-point link).

If the DUPLICATE column is non-empty, then routes from the table named in that column are copied into the new table. By default, all routes (except default routes) are copied. The set of routes copied can be restricted using the COPY column which lists the interfaces whose routes you want copied. You will generally want to include all local interfaces in this list. You should exclude the loopback interface (lo) and any interfaces that do not have an IPv4 configuration. You should also omit interfaces like tun interfaces that are created dynamically. Traffic to networks handled by those interfaces should be routed through the main table using entries in /etc/shorewall/route_rules (see Example 2 below).

In addition:

  1. Unless loose is specified, an ip rule is generated for each IP address on the INTERFACE that routes traffic from that address through the associated routing table.

  2. If you specify track, then connections which have had at least one packet arrive on the interface listed in the INTERFACE column have their connection mark set to the value in the MARK column. In the PREROUTING chain, packets with a connection mark have their packet mark set to the value of the associated connection mark; packets marked in this way bypass any prerouting rules that you create in /etc/shorewall/tcrules. This ensures that packets associated with connections from outside are always routed out of the correct interface.

  3. If you specify balance, then Shorewall will replace the 'default' route with weight 100 in the 'main' routing table with a load-balancing route among those gateways where balance was specified. So if you configure default routes, be sure that their weight is less than 100 or the route added by Shorewall will not be used.

That's all that these entries do. You still have to follow the principle stated in the Shorewall Routing documentation:

  1. Routing determines where packets are to be sent.

  2. Once routing determines where the packet is to go, the firewall (Shorewall) determines if the packet is allowed to go there.

The bottom line is that if you want traffic to go out through a particular provider then you must mark that traffic with the provider's MARK value in /etc/shorewall/tcrules and you must do that marking in the PREROUTING chain; or, you must provide the appropriate rules in /etc/shorewall/route_rules.

/etc/shorewall/masq and Multi-ISP

If you masquerade a local network, you will need to add masquerade rules for both external interfaces. Referring to the diagram above, if each of the interfaces has only a single IP address and you have no systems with public IP addresses behind your firewall, then I suggest the following simple entries:

#INTERFACE       SOURCE            ADDRESS
eth0             0.0.0.0/0         206.124.146.176
eth1             0.0.0.0/0         130.252.99.27

If you have a public subnet (for example 206.124.146.176/30) behind your firewall, then use exclusion:

#INTERFACE       SOURCE               ADDRESS
eth0             !206.124.146.176/29  206.124.146.176
eth1             0.0.0.0/0            130.252.99.27

Note that exclusion is only used on the interface corresponding to internal subnetwork.

If you have multiple IP addresses on one of your interfaces, you can use a similar technique -- simple exclude the smallest network that contains all of those addresses from being masqueraded.

Warning

Entries in /etc/shorewall/masq have no effect on which ISP a particular connection will be sent through. That is rather the purpose of entries in /etc/shorewall/tcrules and /etc/shorewall/route_rules.

Martians

One problem that often arises with Multi-ISP configuration is 'Martians'. If your Internet interfaces are configured with the routefilter option in /etc/shorewall/interfaces (remember that if you set that option, you should also select logmartians), then things may not work correctly and you will see messages like this:

Feb  9 17:23:45 gw.ilinx kernel: martian source 206.124.146.176 from 64.86.88.116, on dev eth1 
Feb  9 17:23:45 gw.ilinx kernel: ll header: 00:a0:24:2a:1f:72:00:13:5f:07:97:05:08:00

The above message is somewhat awkwardly phrased. The source IP in this incoming packet was 64.86.88.116 and the destination IP address is 206.124.146.176. Another gotcha is that the incoming packet has already had the destination IP address changed for DNAT or because the original outgoing connection was altered by an entry in /etc/shorewall/masq (SNAT or Masquerade). So the destination IP address (206.124.146.176) may not have been the destination IP address in the packet as it was initially received.

There a couple of common causes for these problems:

  1. You have connected both of your external interfaces to the same hub/switch. Connecting multiple firewall interfaces to a common hub or switch is always a bad idea that will result in hard-to-diagnose problems.

  2. You are specifying both the loose and balance options on your provider(s). This causes individual connections to ping-pong back and forth between the interfaces which is guaranteed to cause problems.

  3. You are redirecting traffic from the local system out of one interface or the other using packet marking in your /etc/shorewall/tcrules file. A better approach is to configure the application to use the appropriate local IP address (the IP address of the interface that you want the application to use). See below.

If all else fails, remove the routefilter option from your external interfaces. If you do this, you may wish to add rules to log and drop packets from the Internet that have source addresses in your local networks. For example, if the local LAN in the above diagram is 192.168.1.0/24, then you would add this rule:

#ACTION          SOURCE                     DEST
DROP:info        net:192.168.1.0/24         all

Be sure the above rule is added before any other rules with net in the SOURCE column.

Example

The configuration in the figure at the top of this section would be specified in /etc/shorewall/providers as follows.

#NAME   NUMBER  MARK    DUPLICATE       INTERFACE       GATEWAY         OPTIONS          COPY
ISP1    1       1       main            eth0            206.124.146.254 track,balance    eth2
ISP2    2       2       main            eth1            130.252.99.254  track,balance    eth2

Other configuration files go something like this:

/etc/shorewall/interfaces:

#ZONE    INTERFACE    BROADCAST       OPTIONS
net      eth0         detect          …          
net      eth1         detect          …

/etc/shorewall/policy:

#SOURCE    DESTINATION    POLICY     LIMIT:BURST
net        net            DROP

/etc/shorewall/masq (assumes that there are no public subnets behind the firewall):

#INTERFACE       SUBNET            ADDRESS
eth0             0.0.0.0/0        206.124.146.176
eth1             0.0.0.0/0        130.252.99.27

Note

Entries in /etc/shorewall/masq have no effect on which ISP a particular connection will be sent through. That is rather the purpose of entries in /etc/shorewall/tcrules or /etc/shorewall/route_rules.

Routing a Particular Application Through a Specific Interface

This continues the example in the preceding section.

Now suppose that you want to route all outgoing SMTP traffic from your local network through ISP 2. You would make this entry in /etc/shorewall/tcrules (and if you are running a version of Shorewall earlier than 3.0.0, you would set TC_ENABLED=Yes in /etc/shorewall/shorewall.conf).

#MARK           SOURCE          DEST            PROTO   PORT(S) CLIENT  USER    TEST
#                                                               PORT(S)
2:P             <local network> 0.0.0.0/0       tcp     25

Port Forwarding

Shorewall provides considerable flexibility for port forwarding in a multi-ISP environment.

Normal port forwarding rules such as the following will forward from both providers.

/etc/shorewall/rules:

#ACTION        SOURCE             DEST              PROTO     DEST PORT(S)     SOURCE       ORIGINAL
#                                                                              PORTS(S)     DEST
DNAT           net                loc:192.168.1.3   tcp       25

Continuing the above example, to forward only connection requests from ISP 1, you can either:

  1. Qualify the SOURCE by ISP 1's interface:

    #ACTION        SOURCE             DEST              PROTO     DEST PORT(S)     SOURCE       ORIGINAL
    #                                                                              PORTS(S)     DEST
    DNAT           net:eth0           loc:192.168.1.3   tcp       25

    or

  2. Specify the IP address of ISP 1 in the ORIGINAL DEST column:

    #ACTION        SOURCE             DEST              PROTO     DEST PORT(S)     SOURCE       ORIGINAL
    #                                                                              PORTS(S)     DEST
    DNAT           net                loc:192.168.1.3   tcp       25               -            206.124.146.176

More than 2 Providers

When there are more than two providers, you need to extend the two-provider case in the expected way:

  1. For each external address, you need an entry in /etc/shorewall/masq to handle the case where a connection using that address as the SOURCE is sent out of the interfaces other than the one that the address is configured on.

  2. For each external interface, you need to add an entry to /etc/shorewall/masq for each internal network that needs to be masqueraded (or use SNAT) through that interface.

If we extend the above example to add eth3 with IP address 16.105.78.4 with gateway 16.105.78.254, then:

/etc/shorewall/providers:

#NAME   NUMBER  MARK    DUPLICATE       INTERFACE       GATEWAY         OPTIONS          COPY
ISP1    1       1       main            eth0            206.124.146.254 track,balance    eth2
ISP2    2       2       main            eth1            130.252.99.254  track,balance    eth2
ISP3    3       3       main            eth3            16.105.78.254   track,balance    eth2

/etc/shorewall/masq:

#INTERFACE       SUBNET            ADDRESS
eth0             130.252.99.27     206.124.146.176
eth3             130.252.99.27     16.105.78.4
eth1             206.124.146.176   130.252.99.27
eth3             206.124.146.176   16.105.78.4
eth0             16.106.78.4       206.124.146.176
eth1             16.106.78.4       130.252.99.27
eth0             eth2              206.124.146.176
eth1             eth2              130.252.99.27
eth3             eth2              16.105.78.4

Applications running on the Firewall

Experience has shown that in some cases, problems occur with applications running on the firewall itself. This is especially true when you have specified routefilter on your external interfaces in /etc/shorewall/interfaces (see above). When this happens, it is suggested that you have the application use specific local IP addresses rather than 0.

Examples:

  • Squid: In squid.conf, set tcp_outgoing_address to the IP address of the interface that you want Squid to use.

  • In OpenVPN, set local (--local on the command line) to the IP address that you want the server to receive connections on.

Note that some traffic originating on the firewall doesn't have a SOURCE IP address before routing. At least one Shorewall user reports that an entry in /etc/shorewall/route_rules with 'lo' in the SOURCE column seems to be the most reliable way to direct such traffic to a particular ISP.

IPSEC

If you have an IPSEC gateway on your firewall, be sure to arrange for ESP packets to be routed out of the same interface that you have configured your keying daemon to use.

/etc/shorewall/route_rules

The /etc/shorewall/route_rules file was added in Shorewall version 3.2.0. The route_rules file allows assigning certain traffic to a particular provider just as entries in the tcrules file. The difference between the two files is that entries in route_rules are independent of Netfilter.

Routing Rules

Routing rules are maintained by the Linux kernel and can be displayed using the ip rule ls command. When routing a packet, the rules are processed in turn until the packet is successfully routed.

gateway:~ # ip rule ls
0:      from all lookup local                <=== Local (to the firewall) IP addresses
10001:  from all fwmark 0x1 lookup Blarg     <=== This and the next rule are generated by the
10002:  from all fwmark 0x2 lookup Comcast        'MARK' values in /etc/shorewall/providers.
20000:  from 206.124.146.176 lookup Blarg    <=== This and the next rule are generated unless
20256:  from 24.12.22.33 lookup Comcast           'loose' is specified; based in the output of 'ip addr ls'
32766:  from all lookup main                 <=== This is the routing table shown by 'iproute -n'
32767:  from all lookup default              <=== This table is usually empty
gateway:~ #

In the above example, there are two providers: Blarg and Comcast with MARK 1 going to Blarg and mark 2 going to Comcast.

Columns in the route_rules file

Columns in the file are:

SOURCE (Optional)

An ip address (network or host) that matches the source IP address in a packet. May also be specified as an interface name optionally followed by ":" and an address. If the device 'lo' is specified, the packet must originate from the firewall itself.

DEST (Optional)

An ip address (network or host) that matches the destination IP address in a packet.

If you choose to omit either SOURCE or DEST, place "-" in that column. Note that you may not omit both SOURCE and DEST.

PROVIDER

The provider to route the traffic through. May be expressed either as the provider name or the provider number.

PRIORITY

The rule's priority which determines the order in which the rules are processed.

1000-1999 Before Shorewall-generated 'MARK' rules

11000- 11999 After 'MARK' rules but before Shorewall-generated rules for ISP interfaces.

26000-26999 After ISP interface rules but before 'default' rule.

Rules with equal priority are applied in the order in which they appear in the file.

Multi-ISP and VPN

For those VPN types that use routing to direct traffic to remote VPN clients (including but not limited to OpenVPN in routed mode and PPTP), the VPN software adds a host route to the main table for each VPN client. It follows that you must add a routing rule in the 1000-1999 range to specify the main table for traffic addressed to those clients. See Example 2 below.

Examples

Example 1: You want all traffic entering the firewall on eth1 to be routed through Comcast.

#SOURCE            DEST      PROVIDER        PRIORITY
eth1               -         Comcast         1000

With this entry, the output of ip rule ls would be as follows.

gateway:~ # ip rule ls
0:      from all lookup local
1000:   from all iif eth1 lookup Comcast
10001:  from all fwmark 0x1 lookup Blarg
10002:  from all fwmark 0x2 lookup Comcast
20000:  from 206.124.146.176 lookup Blarg
20256:  from 24.12.22.33 lookup Comcast
32766:  from all lookup main
32767:  from all lookup default
gateway:~ #

Note that because we used a priority of 1000, the test for eth1 is inserted before the fwmark tests.

Example 2: You use OpenVPN (routed setup w/tunX) in combination with multiple providers. In this case you have to set up a rule to ensure that the OpenVPN traffic is routed back through the tunX interface(s) rather than through any of the providers. 10.8.0.0/24 is the subnet chosen in your OpenVPN configuration (server 10.8.0.0 255.255.255.0).

#SOURCE                 DEST            PROVIDER        PRIORITY
-                       10.8.0.0/24     main            1000

USE_DEFAULT_RT

Beginning with Shorewall 4.2.0 Beta3, Shorewall-perl has supported a USE_DEFAULT_RT option in shorewall.conf (5).

One of the drawbacks of the Multi-ISP support as described in the preceding sections is that changes to the main table made by applications are not added to the individual provider tables. This makes route rules such as described in one of the examples above necessary.

USE_DEFAULT_RT=Yes works around that problem by passing packets through the main table first rather than last. This has a number of implications:

  1. Both the DUPLICATE and the COPY columns in the providers file must remain empty or contain "-". The individual provider routing tables generated when USE_DEFAULT_RT=Yes contain only a host route to the gateway and a default route via the gateway.

  2. The balance option is assumed for all interfaces that do not have the loose option. When you want both balance and loose, both must be specified.

  3. The default route generated by Shorewall is added to the default routing table (253) rather than to the main routing table (254).

  4. Packets are sent through the main routing table by a routing rule with priority 999. In ), the priority range 1-998 may be used for inserting rules that bypass the main table.

  5. You should disable all default route management outside of Shorewall. If a default route is inadvertently added to the main table while Shorewall is started, then all policy routing will stop working except for those routing rules in the priority range 1-998.

  6. Because all default route management is disabled, Shorewall is unable to use the main routing table to detect gateway addresses.

    • For ppp interfaces, the GATEWAY may remain unspecified ("-").

    • Beginning with Shorewall 4.2.6, 'detect' may be specified for interfaces whose configuration is managed by dhcpcd. Shorewall will use dhcpcd's database to determine the gateway IP address.

    • Beginning with Shorewall 4.2.8, 'detect' may be specified for interfaces whose configuration is managed by dhclient. Shorewall will use dhclient's database to determine the gateway IP address.

    • All other interfaces must have an IP address specified in the GATEWAY column.

Although 'balance' is automatically assumed when USE_DEFAULT_RT=Yes, you can easily cause all traffic to use one provider except when you explicitly direct it to use the other provider via shorewall-route_rules (5) or shorewall-tcrules (5).

Example (send all traffic through the 'shorewall' provider unless otherwise directed).

/etc/shorewall/providers:

#NAME    NUMBER MARK DUPLICATE INTERFACE GATEWAY       OPTIONS
linksys       1    1    -        wlan0   172.20.1.1    track,balance=1,optional
shorewall     2    2    -        eth0    192.168.1.254 track,balance=2,optional

/etc/shorewall/route_rules:

#SOURCE     DEST      PROVIDER        PRIORITY
-           -         shorewall       11999

Gateway Monitoring and Failover

There are a couple of options available for monitoring the status of provider links and taking action when a failure occurs.

When using either solution, it is important that the providers be optional.

If you are using Shorewall 4.0, you must specify the optional option in /etc/shorewall/providers:

#NAME   NUMBER  MARK    DUPLICATE       INTERFACE       GATEWAY         OPTIONS                   COPY
ISP1    1       1       main            eth0            206.124.146.254 track,balance,optional    eth2
ISP2    2       2       main            eth1            130.252.99.254  track,balance,optional    eth2

If you are using Shorwall 4.2 (which is highly recommended), you specify the optional option in /etc/shorewall/interfaces:

#ZONE    INTERFACE    BROADCAST       OPTIONS
net      eth0         detect          optional         
net      eth1         detect          optional

SWPING

Beginning with Shorewall 4.2.6, Shorewall includes a sample monitoring script swping. The swping file is available in the main directory contained in the Shorewall-common tarball and is included in the Shorewall-common documentation directory on the Shorewall-common RPM. The script is inspired by Angsuman Chakraborty's gwping script.

For those not on 4.2.6 yet, the script may be downloaded from http://www.shorewall.net/pub/shorewall/contrib/MultiISP-failover/.

Important

These samples are offered as is — they work for me but I don't make any claim that they will work for anyone else. But if you have a need for automated link monitoring, they offer you a place to start.

The script should be copied to a directory on root's PATH such as /usr/local/sbin/.

The script works by sending pings to target IP addresses through each external interface. These targets must not depend on any routes other than those that are present in the main routing table. That ensures that a route is available to the target even when the target's interface is not working and Shorewall has omitted it from the routing configuration. An interface is assumed to be up when a specified number (UP_COUNT) of consecutive ping operations succeed. Similarly, an interface is assumed to be down when a specified number (DOWN_COUNT) of consecutive ping operations fail. You can specify the interval between pings (PING_INTERVAL).

The script monitors two interfaces but it is a trivial exercise to extend it to more than two. At the top are a number of variables to set:

#
# IP family -- 4 or 6
#
FAMILY=4
#
# The commands to run when the status of a line changes. Multiple commands may be specified 
# when separated by semicolons (";")
#
COMMAND=

...
#
# Interfaces to monitor -- you may use shell variables from your params file
#
IF1=eth0
IF2=eth1
#
# Sites to Ping. Must depend only on routes in the 'main' routing table. If not specified,
# the interface is assumed to be managed by dhcpcd and the script uses the gateway address
# from /var/lib/dhcpcd/dhcpcd-${IFx}.info
#
TARGET1=
TARGET2=
#
# How often to ping
#
PING_INTERVAL=5
#
# Value for ping's -W option
#
PING_TIMEOUT=2
#
# This many successive pings must succeed for the interface to be marked up when it is down
#
UP_COUNT=5
#
# This many successive pings must fail for the interface to be marked down when it is up
#
DOWN_COUNT=2

If you leave COMMAND empty, the script sets its value automatically depending on whether Shorewall-lite is installed.

When the status of an interface changes:

  • For each interface, a file is placed in /etc/shorewall to record the status of the interface: either 0 (UP) or 1 (DOWN). The name of the file is interface.status where interface is the interface (e.g., eth0.status).

  • A shorewall -f restart command is executed (shorewall-lite restart, if Shorewall-lite is installed).

  • The contents of the main routing table are displayed.

The .status files are intended to be used with the following /etc/shorewall/isusable script.

local status=0

case $1 in
    eth0|eth1)
        [ -f /etc/shorewall/${1}.status ] && status=$(cat /etc/shorewall/${1}.status)
        ;;
esac

return $status

Be sure that you modify the interface names to match your configuration.

Also included is a sample init script (swping.init) to start the monitoring daemon. Copy it to /etc/init.d/swping and use your distribution's SysV init tools to cause it to be run at boot. It works on OpenSuSE™ 11.0 -- YMMV. Modify the PROG and STATEDIR variables as needed.

As an alternative to using the init script, you can add the following to /etc/shorewall/started:

if [ "$COMMAND" = start ]; then
   killall -9 swping 2> /dev/null #be sure that there are none left running
   /usr/local/sbin/swping &
fi

and add this to /etc/shorewall/stopped.

if [ "$COMMAND" = stop -o "$COMMAND" = clear ]; then
   killall -9 swping 2> /dev/null
fi

This simple script has a number of limitations:

  1. It only works on IPv4 or IPv6 but not both at once. So if you want to monitor both IPv4 and IPv6, you need to clone the script are run two copies; one for IPv4 and one for IPv6.

  2. It can only detect the gateway for interfaces managed by dhcpcd.

  3. It's method of determining whether an interface is up or down is crude. You will normally specify the default gateway for each provider as the sites to ping and being able to ping the default gateway is not a surefire indication that the provider is usable. The method of determining whether a site is up or down is also crude.

  4. Because of the crudeness of the algorithm, hysteresis may occur.

  5. It is tricky to configure a system such that the system works correctly when one of its providers is down unless you largely don't care which interface is used.

Link Status Monitor (LSM)

Link Status Monitor was written by Mika Ilmaranta <ilmis at nullnet.fi> and performs more sophisticated monitoring than the simple swping script described in the preceding section.

I personally use LSM here at shorewall.net (configuration is described below). I have set things up so that Shorewall [re]starts lsm during processing of the start and restore commands. I don't have Shorewall restart lsm during Shorewall restart because I restart Shorewall much more often than the average user is likely to do. I have Shorewall start lsm because I have a dynamic IP address from one of my providers (Comcast); Shorewall detects the default gateway to that provider and creates a secondary configuration file (/etc/lsm/shorewall.conf) that contains the link configurations. That file is included by /etc/lsm/lsm.conf.B

Below are my relevant configuration files.

Warning

These files only work with Shorewall-perl 4.2 and later.

/etc/shorewall/isusable:

local status
status=0

[ -f /etc/shorewall/${1}.status ] && status=$(cat /etc/shorewall/${1}.status)

return $status

/etc/shorewall/started:

###############################################################################
# My 'restored' script calls this one if there is no lsm process running
###############################################################################
if [ "$COMMAND" = start -o "$COMMAND" = restore ]; then
   killproc lsm 2> /dev/null
   cat <<EOF > /etc/lsm/shorewall.conf
connection {
    name=Avvanta
    checkip=206.124.146.254
    device=eth0
    ttl=2
}

connection {
    name=Comcast
    checkip=${ETH3_GATEWAY:-71.227.156.1}
    device=eth3
    ttl=1
}
EOF
   rm -f /etc/shorewall/*.status
   /usr/sbin/lsm /etc/lsm/lsm.conf >> /var/log/lsm
fi

eth3 has a dynamic IP address so I need to use the Shorewall-detected gateway address ($ETH3_GATEWAY).

/etc/shorewall/restored:

if [ -z "$(ps ax | grep 'lsm ' | grep -v 'grep ' )" ]; then
   run_started_exit
fi

/etc/lsm/lsm.conf:

#
# Defaults for the connection entries
#
defaults {
  name=defaults
  checkip=127.0.0.1
  eventscript=/etc/lsm/script
  max_packet_loss=20
  max_successive_pkts_lost=7
  min_packet_loss=5
  min_successive_pkts_rcvd=10
  interval_ms=2000
  timeout_ms=2000
  warn_email=teastep@shorewall.net
  check_arp=0
  sourceip=
  ttl=64
}

include /etc/lsm/shorewall.conf

/etc/lsm/script

#!/bin/sh
#
# (C) 2009 Mika Ilmaranta <ilmis@nullnet.fi>
# (C) 2009 Tom Eastep <teastep@shorewall.net>
#
# License: GPLv2
#

STATE=${1}
NAME=${2}
CHECKIP=${3}
DEVICE=${4}
WARN_EMAIL=${5}
REPLIED=${6}
WAITING=${7}
TIMEOUT=${8}
REPLY_LATE=${9}
CONS_RCVD=${10}
CONS_WAIT=${11}
CONS_MISS=${12}
AVG_RTT=${13}

cat <<EOM | mail -s "${NAME} ${STATE}, DEV ${DEVICE}" ${WARN_EMAIL}

Hi,

Connection ${NAME} is now ${STATE}.

Following parameters were passed:
newstate     = ${STATE}
name         = ${NAME}
checkip      = ${CHECKIP}
device       = ${DEVICE}
warn_email   = ${WARN_EMAIL}

Packet counters:
replied      = ${REPLIED} packets replied
waiting      = ${WAITING} packets waiting for reply
timeout      = ${TIMEOUT} packets that have timed out (= packet loss)
reply_late   = ${REPLY_LATE} packets that received a reply after timeout
cons_rcvd    = ${CONS_RCVD} consecutively received replies in sequence
cons_wait    = ${CONS_WAIT} consecutive packets waiting for reply
cons_miss    = ${CONS_MISS} consecutive packets that have timed out
avg_rtt      = ${AVG_RTT} average rtt, notice that waiting and timed out packets have rtt = 0 when calculating this

Your LSM Daemon

EOM

[ ${STATE} = up ] && state=0 || state=1

echo $state > /etc/shorewall/${DEVICE}.status

/sbin/shorewall -f restart >> /var/log/lsm 2>&1

/sbin/shorewall show routing >> /var/log/lsm

exit 0;

#EOF

:

Two Providers Sharing an Interface

Shared interface support is available only in Shorewall-perl 4.2.0 and later.

  1. Only Ethernet (or Ethernet-like) interfaces can be used. For inbound traffic, the MAC addresses of the gateway routers are used to determine which provider a packet was received through. Note that only routed traffic can be categorized using this technique.

  2. You must specify the address on the interface that corresponds to a particular provider in the INTERFACE column by following the interface name with a colon (":") and the address.

  3. Entries in /etc/shorewall/masq must be qualified by the provider name (or number).

  4. This feature requires Realm Match support in your kernel and iptables.

  5. You must add route_rules entries for networks that are accessed through a particular provider.

  6. If you have additional IP addresses through either provider, you must add route_rules to direct traffic FROM each of those addresses through the appropriate provider.

  7. You must manually add MARK rules for traffic known to come from each provider.

  8. You must specify a gateway IP address in the GATEWAY column of /etc/shorewall/providers; detect is not permitted.

Taken together, b. and h. effectively preclude using this technique with dynamic IP addresses.

Example:

This is our home network circa fall 2008. We have two Internet providers:

  1. Comcast -- Cable modem with one dynamic IP address.

  2. Avvanta -- ADSL with 5 static IP addresses.

Because the old CompaqPresario™ that I use for a firewall only has three PCI slots and no onboard Ethernet, it doesn't have enough Ethernet controllers to support both providers. So I use a Linksys WRT300n pre-N router as a gateway to Comcast. Note that because the Comcast IP address is dynamic, I could not share a single firewall interface between the two providers directly.

On my personal laptop (ursa), I have 9 virtual machines running various Linux distributions. It is the Shorewall configuration on ursa that I will describe here.

Below is a diagram of our network:

The local wired network in my office is connected to both gateways and uses the private (RFC 1918) network 172.20.1.0/24. The Comcast gateway has local IP address 172.20.1.1 while the Avvanta gateway has local IP address 172.20.1.254. Ursa's eth0 interface has a single IP address (172.20.1.130).

This configuration uses USE_DEFAULT_RT=Yes in shorewall.conf (see above).

Here is the providers file:

#NAME          NUMBER   MARK DUPLICATE INTERFACE            GATEWAY      OPTIONS                            COPY
comcast        1        1    -         eth0:172.20.1.130    172.20.1.1   track,loose,balance,optional
avvanta        2        2    -         eth0:172.20.1.130    172.20.1.254 track,optional,loose
wireless       3        3    -         wlan0                172.20.1.1   track,optional

Several things to note:

  1. 172.20.1.130 is specified as the eth0 IP address for both providers.

  2. Both wired providers have the loose option. This prevents Shorewall from automatically generating routing rules based on the source IP address.

  3. Only comcast has the balance option. With USE_DEFAULT_RT=yes, that means that comcast will be the default provider. While balance is the default, with USE_DEFAULT_RT=Yes, it must be specified explicitly when loose is also specified.

  4. I always disable the wireless interface when the laptop is connected to the wired network.

  5. I use a different Shorewall configuration when I take the laptop on the road.

Here is the route_rules file:

#SOURCE                 DEST                    PROVIDER        PRIORITY
-                       206.124.146.176/31      avvanta         1000
-                       206.124.146.178/31      avvanta         1000
-                       206.124.146.180/32      avvanta         1000

Those rules direct traffic to the five static Avvanta IP addresses (only two are currently used) through the avvanta provider.

Here is the tcrules file (MARK_IN_FORWARD_CHAIN=No in shorewall.conf):

#MARK           SOURCE          DEST            PROTO   PORT(S)         CLIENT  USER    TEST    LENGTH  TOS     CONNBYTES       HELPER
#                                                                       PORT(S) 
2               $FW             0.0.0.0/0       tcp     21
2               $FW             0.0.0.0/0       tcp     -               -       -       -       -       -       -               ftp
2               $FW             0.0.0.0/0       tcp     119

These rules:

  • Use avvanta for FTP.

  • Use avvanta for NTTP

The remaining files are for a rather standard two-interface config with a bridge as the local interface.

zones:

#ZONE   IPSEC   OPTIONS                 IN                      OUT
#       ONLY                            OPTIONS                 OPTIONS
fw      firewall
net     ipv4
kvm     ipv4

policy:

net             net             NONE
fw              net             ACCEPT
fw              kvm             ACCEPT
kvm             all             ACCEPT
net             all             DROP            info
all             all             REJECT          info

interfaces:

#ZONE    INTERFACE      BROADCAST       OPTIONS                 GATEWAY
#
net     eth0            detect          dhcp,tcpflags,routefilter,blacklist,logmartians,optional,arp_ignore
net     wlan0           detect          dhcp,tcpflags,routefilter,blacklist,logmartians,optional
kvm     br0             detect          routeback       #Virtual Machines

Note

wlan0 is the wireless adapter in the notebook. Used when the laptop is in our home but not connected to the wired network.

masq:

#INTERFACE              SUBNET          ADDRESS         PROTO   PORT(S) IPSEC
eth0                    192.168.0.0/24
wlan0                   192.168.0.0/24

Note

Because the firewall has only a single external IP address, I don't need to specify the providers in the masq rules.

A Complete Working Example

This section describes the network at shorewall.net early in 2009. The configuration is as follows:

  • Two providers:

    • Avvanta -- A slow (1.5mb/384kb) DSL service with 5 static IP addresses.

    • Comcast -- A fast (20mb/10mb) Cable circuit with a single dynamic address.

  • A local network consisting of wired and wireless client systems. A Linksys WRT300N wireless router is used as an access point for the wireless hosts.

  • A DMZ hosting a single server (lists.shorewall.net aka www1.shorewall.net, ftp1.shorewall.net,etc.)

The network is pictured in the following diagram:

Because of the speed of the cable provider, all traffic uses that provider unless there is a specific need for the traffic to use the DSL line.

  • Responses to connections from the Internet to one of the DSL IP addresses -- the track option takes care of that.

  • Connections initiated by the server and connections requested by clients on the firewall that have bound their local socket to one of the DSL IP addresses. Two entries in /etc/shorewall/route_rules take care of that traffic.

As a consequence, I have disabled all route filtering on the firewall and only use the balance option in /etc/shorewall/providers on the Comcast provider whose default route in the main table is established by DHCP. By specifying the fallback option on Avvanta, I ensure that there is still a default route if Comcast is down. lsm is used to monitor the links.

/etc/sysctl.conf:

net.ipv4.conf.all.rp_filter = 0

/etc/shorewall/shorewall.conf:

ROUTE_FILTER=No
RESTORE_DEFAULT_ROUTE=No

The RESTORE_DEFAULT_ROUTE option was added in Shorewall-perl 4.2.6 and causes the default route in the main table to be deleted when the Comcast link is unavailable. That way, the default route in the default table will be used until Comcast is available again.

/etc/shorewall/providers:

#NAME  NUMBER MARK  DUPLICATE INTERFACE GATEWAY         OPTIONS                 COPY
Avvanta     1 0x100 main      eth0      206.124.146.254 track,loose,fallback    eth2,eth4,tun*
Comcast     2 0x200 main      eth3      detect          track,balance           eth2,eth4,tun*
#LAST LINE -- ADD YOUR ENTRIES ABOVE THIS LINE -- DO NOT REMOVE

The loose option on Avvanta results in fewer routing rules. The first two routing rules below insure that all traffic from Avvanta-assigned IP addresses is sent via the Avvanta provider. The 'tun*' included in the COPY column is there because I run a routed OpenVPN server on the firewall.

/etc/shorewall/route_rules:

#SOURCE            DEST           PROVIDER PRIORITY
-                  172.20.0.0/24  main     1000    # Addresses assigned by routed OpenVPN server
206.124.146.176/30 -              Avvanta  26000
206.124.146.180    -              Avvanta  26000
-                  216.168.3.44   Avvanta  26000   # Avvanta NNTP Server -- verifies source IP address
#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE

The /etc/shorewall/route_rules entries provide all of the provider selection necessary so my /etc/shorewall/tcrules file is used exclusively for traffic shaping of the Avvanta line. Note that I still need to provide values in the MARK colum of /etc/shorewall/providers because I specify track on both providers.

Here is the output of shorewall show routing:

Routing Rules

0:     from all lookup local
1000:  from all to 172.20.0.0/24 lookup main 
10000: from all fwmark 0x100 lookup Avvanta 
10001: from all fwmark 0x200 lookup Comcast 
20256: from 71.227.156.229 lookup Comcast 
26000: from 206.124.146.176/30 lookup Avvanta 
26000: from 206.124.146.180 lookup Avvanta 
26000: from all to 216.168.3.44 lookup Avvanta 
32766: from all lookup main 
32767: from all lookup default 

Table Avvanta:

206.124.146.254 dev eth0  scope link  src 206.124.146.176 
206.124.146.177 dev eth4  scope link 
172.20.1.0/24 dev eth2  proto kernel  scope link  src 172.20.1.254 
206.124.146.0/24 dev eth0  proto kernel  scope link  src 206.124.146.176 
169.254.0.0/16 dev eth0  scope link 
default via 206.124.146.254 dev eth0  src 206.124.146.176 

Table Comcast:

206.124.146.177 dev eth4  scope link 
71.227.156.1 dev eth3  scope link  src 71.227.156.229 
172.20.1.0/24 dev eth2  proto kernel  scope link  src 172.20.1.254 
71.227.156.0/23 dev eth3  proto kernel  scope link  src 71.227.156.229 
default via 71.227.156.1 dev eth3  src 71.227.156.229 

Table default:

default via 206.124.146.254 dev eth0 metric 1

Table local:

broadcast 127.255.255.255 dev lo  proto kernel  scope link  src 127.0.0.1 
broadcast 172.20.1.0 dev eth2  proto kernel  scope link  src 172.20.1.254 
broadcast 206.124.146.255 dev eth0  proto kernel  scope link  src 206.124.146.176 
local 206.124.146.179 dev eth0  proto kernel  scope host  src 206.124.146.176 
local 206.124.146.178 dev eth0  proto kernel  scope host  src 206.124.146.176 
local 206.124.146.176 dev eth0  proto kernel  scope host  src 206.124.146.176 
local 206.124.146.176 dev eth4  proto kernel  scope host  src 206.124.146.176 
broadcast 71.227.157.255 dev eth3  proto kernel  scope link  src 71.227.156.229 
broadcast 71.227.156.0 dev eth3  proto kernel  scope link  src 71.227.156.229 
local 172.20.1.254 dev eth2  proto kernel  scope host  src 172.20.1.254 
local 127.0.0.2 dev lo  proto kernel  scope host  src 127.0.0.1 
broadcast 172.20.1.255 dev eth2  proto kernel  scope link  src 172.20.1.254 
local 71.227.156.229 dev eth3  proto kernel  scope host  src 71.227.156.229 
broadcast 206.124.146.0 dev eth0  proto kernel  scope link  src 206.124.146.176 
broadcast 127.0.0.0 dev lo  proto kernel  scope link  src 127.0.0.1 
local 206.124.146.180 dev eth0  proto kernel  scope host  src 206.124.146.176 
local 127.0.0.1 dev lo  proto kernel  scope host  src 127.0.0.1 
local 127.0.0.0/8 dev lo  proto kernel  scope host  src 127.0.0.1 

Table main:

206.124.146.177 dev eth4  scope link 
172.20.1.0/24 dev eth2  proto kernel  scope link  src 172.20.1.254 
206.124.146.0/24 dev eth0  proto kernel  scope link  src 206.124.146.176 
71.227.156.0/23 dev eth3  proto kernel  scope link  src 71.227.156.229 
169.254.0.0/16 dev eth0  scope link 
127.0.0.0/8 dev lo  scope link 
default via 71.227.156.1 dev eth3 

/etc/shorewall/interfaces:

#ZONE    INTERFACE         BROADCAST           OPTIONS
loc      eth2              detect              dhcp,routeback
dmz      eth4              detect
net      eth0              detect              dhcp,blacklist,tcpflags,optional
net      eth3              detect              dhcp,blacklist,tcpflags,optional
#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE

/etc/shorewall/masq:

#INTERFACE   SOURCE           ADDRESS      PROTO      PORT(S)      IPSEC

COMMENT Masquerade Local Network
eth3         0.0.0.0/0
eth0        !206.124.146.0/24 206.124.146.179

#LAST LINE -- ADD YOUR ENTRIES ABOVE THIS LINE -- DO NOT REMOVE

All traffic leaving eth3 must use the dynamic IP address assigned to that interface as the SOURCE address. All traffic leaving eth0 that does not have a SOURCE address falling within the Avvanta subnet (206.124.146.0/24) must have its SOURCE address changed to 206.124.146.179.