Lewis' Blog Tales from the trenches of information technology

Multiple default routes / public gateway IPs under Linux

I recently had the need to configure a server for a client with multiple public IPv4 addresses routing to the internet and the requirement to switch between them at will while browsing (http traffic only).

There are a number of articles available on the net dealing with this type of situation, mainly focused on using iproute2, ToS tagging, and Squid (see references, below). However, I bumped into an issue with openSUSE 12.1 where it would stubbornly refuse to accept certain (otherwise valid) ToS (DSCP) values (see https://bugzilla.novell.com/show_bug.cgi?id=770785 for my bug report). This severely limited the number of possible values I could use, and thus, the number of possible public IP addresses.

Luckily, Squid 3 seems to have beefed up the usefulness of the tcp_outgoing_address directive, and with this, it is possible to direct Squid's traffic without using any ToS values or fwmarks (another possible solution I considered; I'll post more about both of these IP functions in a later article).

The main steps to my solution are:

1. Add virtual IP addresses to a given interface (I used YaST in openSUSE, but there are obviously several ways to accomplish this).
2. Add tables to /etc/iproute2/rt_tables for the various virtual networks associated with the virtual IPs:

1. Open /etc/iproute2/rt_tables in your favorite editor;
2. To the end of the file, append entries, one per line, for each new routing table you wish to add, e.g.:

   1000 table1
   2000 table2

3. Save the file.

3. Add rules for traffic flowing to and from each virtual IP, as well as routes for each:

   ip rule add from xxx.xxx.xxx.xxx/32 table t1
   ip rule add to xxx.xxx.xxx.xxx/32 table t1
   
   ip rule add from yyy.yyy.yyy.yyy/32 table t2
   ip rule add to yyy.yyy.yyy.yyy/32 table t2
   
   ip route add xxx.xxx.xxx.0/24 dev eth0 src xxx.xxx.xxx.xxx table t1
   ip route add yyy.yyy.yyy.0/24 dev eth0 src yyy.yyy.yyy.yyy table t2
   
   ip route add default via xxx.xxx.xxx.xxx table t1
   ip route add default via yyy.yyy.yyy.yyy table t1

(Apparently, /24 nets seem to work better with iproute2 - there have been reports of rules getting ignored for smaller subnets. I haven't tested this, though.)

4. Add ACLs to squid.conf for each virtual IP:

   acl test1 src xxx.xxx.xxx.xxx/32
   acl test2 src yyy.yyy.yyy.yyy/32

5. Add http_access rules to squid.conf for each virtual IP (if necessary):

   http_access allow table1
   http_access allow table2

6. Add http_port directives to squid.conf for each virtual IP and listening port:

   http_port xxx.xxx.xxx.xxx:3128
   http_port yyy.yyy.yyy.yyy:3128

7. Add tcp_outgoing_address directives to squid.conf for each virtual IP bound to its respective ACL:

   tcp_outgoing_address xxx.xxx.xxx.xxx table1
   tcp_outgoing_address yyy.yyy.yyy.yyy table2

   (Restarting/reloading Squid may be required if not bouncing the entire box following the balance of the configuration.)

8. Put the policy routing rules in some script to be processed after the intended interface comes up (on openSUSE, this can be in /etc/init.d/after.local - for a single interface which does not go up and down during normal operation, or in a post-up script for a given interface - see here and here for details and methods to enable after.local under openSUSE 12.1 and above; on other distros, rc.local might be a good choice, or of course, for situations where these rules need to be applied only when a certain interface has come up, utilizing a post-up script should do). The thing to remember is that without adding these rules to some automatically started script, they will *not* persist across reboots.
9. Configure the browser for each listening address:port for Squid. I used Proxy Selector  under Firefox, and Proxy Switchy! for Chromium.

There are further steps required if behind a firewall, as it is necessary to then route all incoming traffic from a given private IP/net to a given public IP. I'll go into that part of the configuration in a later article. For now, though, using the above, it seems to be possible to create as many outgoing addresses as may be required, with the ability to switch between them easily.

From a terminal, it's also possible to utilize these extra IP's for testing purposes. For example:

ping -I xxx.xxx.xxx.xxx aaa.aaa.aaa.aaa

and

traceroute -S xxx.xxx.xxx.xxx bbb.bbb.bbb.bbb

and so forth (for commands which allow specifying a source address).

References:

Linux Advanced Routing & Traffic Control HOWTO (appears to be offline as of this edit, 2018-07-25))
Linux Advanced Routing & Traffic Control HOWTO (alternate mirror at The Linux Documentation Project)
HOWTO: Multirouting with Linux
Multiple outgoing addresses for Squid? [linux multihoming gateway]
Multi gateway routing with iptables and iproute2
Configuring Multiple Default Routes in Linux
Selective load balancing with Squid and iproute2 (utilizes ToS field for differentiating traffic)
The Quick Guide to LARTC plus five alternative methods (suggested by the publisher 2018-07-25; looks like good info)

Last Updated on July 25th, 2018 by LewisR