Multiple Default Gateways under Linux with iproute2
Under Linux, it's possible to configure your default route on a process-by-process basis, and hence use multiple interfaces to reach a single end-point. For example, you can connect multiple modems to one host, and simultaneously use these, quite independently, to hit a single remote site. This has important applications in comparitive analysis of networks; for example in performance evaluation. I've also heard of people configuring squid using techniques similar to this for quite specific purposes, but I've only provided details of my own configuration.
The setup for multiple default gateways I'm describing does NOT load-balance multiple connections stochastically. To each remote host it appears that all of the connections are independent. I have not come across another guide which explains how to do this successfully; my results were obtained largely by trying things out and debugging.
The example is provided 'as-is'; this is not a HOWTO, more a "How I did it". It works for me, and hopefully I can point out a couple of the gotchas that slowed me down. Hopefully it's useful to the Linux community.
Configuring static files for iproute2
As I said, we're creating multiple default routes. But which packet belongs on which route? Simple, we'll set a mark in the metadata for the packet. Make the following entries in your iproute2 configuration files. The effect of this is to give ourselves some friendly names (connection1, connection2, etc) to refer to our different routing tables by./etc/iproute2/rt_tables: # # reserved values # 255 local 254 main 253 default 0 unspec 151 connection1 152 connection2 153 connection3 154 connection4 # # local # 1 inr.ruhep
Use the ip tool to mark data for the multiple default routes
The association between the routing table named in the previous section, and the mark set in the packet, is determined by the ip tool like so;ip rule add fwmark 1 table connection1 ip rule add fwmark 2 table connection2
Mark the packets in the output chain with an appropriate mark
Clearly we could use any criterion for marking the packets. In my case, I needed to filter by uid, so I've provided that; adjust to your own taste.
iptables -t mangle -A OUTPUT -m owner --uid-owner 500 -j MARK --set-mark 1
Add a SNAT rule to take care of the IP address
You will need to change the source address of packets you are sending, since they're now (most likely) stamped with a source address which is different from the interface they're going out on.
iptables -t nat -A POSTROUTING -o YOURINTERFACE \
-j SNAT --to-source=YOURINTERFACE_IP
In the case of a PPP connection with dynamic addresses, you won't know the correct IP address to SNAT to, so you will need to write a small script to take care of this.
Set your multiple default routes; each interface by table
You're now ready to associate the routes you've created with a physical device. It works just like normal, except for you'll tell it which table's default route you're setting, whereas historically you've probably only messed around with the 'main' table.ip route add default dev YOURINTERFACE1 table connection1 ip route add default dev YOURINTERFACE2 table connection2
Gotchas
Disable rp_filter
Why aren't the packets coming back? I was scratching my head for a little while with this one. Correctly addressed packets going out the interface, yet nothing coming back.
Setting the default route of the main table to point out the interface instantly fixes this, but that sort of defeats the purpose really, doesn't it? If I've understood correctly, this problem happens because the test done on incoming packets as to whether they are bogus is applied from the main table, and no others. Your main table will point to at most one place, so the others won't, by default, work. Here's how you fix it:echo 0 > /proc/sys/net/ipv4/conf/YOURINTERFACE/rp_filter
Doesn't send packets
Sometimes (particularly in the case where a gethostbyname() is involved) the process just can't seem to start sending packets. I never figured out exactly the cause of all this, but I suspected it was because I didn't have a correct default route on my main table.
In any event, the remedy was to instantiate a dummy device and point the default route at it. No packets should ever hit the dummy0 device, but having it there makes it all work. Your mileage may vary.
ifconfig dummy0 1.2.3.4 ip route add default dev dummy0
It solved my problem, though I'd love to hear if anybody knows why.