tech.ssut

Using Multiple NICs on Linux

Suhun Han -

I’m running a Linux machine having multiple NICs, one for wired connection, one for WWAN(Wireless WAN). The latter is connected to the cellular network using Sierra Wireless LTE modem with Qualcomm Snapdragon X55 Modem and is for scrapping tasks. The reason I use this structure is that some websites implemented intelligent anti-crawling mechanisms thus repetitive requests will be likely detected and banned. So I believed cellular network can deal with this issue easily because it's used by so many devices around in a small cell but also because of its dynamic IP assignments – The external IP will be randomly changed as time goes on.

At first, I thought it would be easy to set up. I tried to run a container with a network driver created using brctl and iptables' masquerade but it failed because there was one problem that I've forgotten: Routing Tables. (Note: Of course I did under the condition IP forwarding is enabled.)

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: wwp42s0f3u2i12: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq state UNKNOWN group default qlen 1000
    inet 10.181.223.215/28 brd 10.181.223.223 scope global noprefixroute wwp42s0f3u2i12
       valid_lft forever preferred_lft forever
4: enp35s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 10.100.0.230/24 brd 10.100.0.255 scope global dynamic noprefixroute enp35s0f0
       valid_lft 48434sec preferred_lft 48434sec
8: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
NIC Setup

Because gateway routing is not so smart enough - maybe I was - that the answer is not sent back via the same route a request sent the packet will effectively lose when I send a request using the second NIC that has its own gateway.

Then I changed the plan to set up a SOCKS proxy server using Dante because it supports specifying an outgoing network interface.

Ookla SpeedTest CLI supports specifying network interface. wwp42s0f3u2i12 is an interface connected to cellular network.

This one with Dante seems like it could be easy but it wasn't because the same problem happened. Dante support specifying the address to be used for outgoing connections but the packet didn't traverse as intended, just port unreachable starts to appear in tcpdump.

Well, I switched the plan again and decided to go with iptables' mangle, which is used to modify or mark packets. In order to achieve this with Dante, I created a new user 'proxy' and then make all network traffic established by this user to go through the wwp42s0f3u2i12 interface, which has a dynamically assigned internal IP. The 'owner' module of iptables can be used to match packets from specific users, and iproute2 can be used for marking outgoing packets when they arrive at the firewall. So I added another routing table called ltegw(201) which will be used for packets marked 0x1:

$ cat /etc/iproute2/rt_tables
201 ltegw

And added ip rule so all marked packets will go to the new routing table:

$ sudo ip rule add from all fwmark 1 table 201
$ ip rule
1:	from all lookup local
32765:	from all fwmark 0x1 lookup 201
32766:	from all lookup main
32767:	from all lookup default
$ sudo ip route add 0.0.0.0/0 dev wwp42s0f3u2i12 table 200
$ ip r
default via 10.100.0.1 dev enp35s0f0 proto dhcp metric 101
default via 10.181.223.216 dev wwp42s0f3u2i12 proto static metric 700
10.100.0.0/24 dev enp35s0f0 proto kernel scope link src 10.100.0.230 metric 101
10.181.223.208/28 dev wwp42s0f3u2i12 proto kernel scope link src 10.181.223.215 metric 700
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown

Next, I added a new iptables OUTPUT rule to mangle table so that packets from UID 1001 (proxy) are marked with 0x1 as they pass through the firewall:

$ sudo iptables -A OUTPUT -t mangle -j MARK --set-mark 1 -m owner --uid-owner proxy
$ sudo iptables -L OUTPUT -t mangle --line-numbers
Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination
1    MARK       all  --  anywhere            !anywhere             owner UID match proxy MARK set 0x1
2    MARK       all  --  anywhere             anywhere             owner UID match proxy MARK set 0x

After this, I see requests from the proxy user get sent on wwp42s0f3u2i12 and a response is successfully received:

$ docker run --rm -it -u 1001:1001 --network host alpine wget -q -O - ifconfig.io/ip
175.223.10.145

# simultaneous session
$ sudo tcpdump -nni wwp42s0f3u2i12
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wwp42s0f3u2i12, link-type EN10MB (Ethernet), capture size 262144 bytes
09:00:01.409730 IP 10.181.223.215.54007 > 211.246.100.49.53: 26241+ A? ifconfig.io. (29)
09:00:01.409740 IP 10.181.223.215.54007 > 168.126.63.1.53: 26241+ A? ifconfig.io. (29)
09:00:01.409745 IP 10.181.223.215.54007 > 10.100.0.1.53: 26241+ A? ifconfig.io. (29)
09:00:01.409748 IP 10.181.223.215.54007 > 211.246.100.49.53: 26431+ AAAA? ifconfig.io. (29)
09:00:01.409750 IP 10.181.223.215.54007 > 168.126.63.1.53: 26431+ AAAA? ifconfig.io. (29)
09:00:01.409752 IP 10.181.223.215.54007 > 10.100.0.1.53: 26431+ AAAA? ifconfig.io. (29)
09:00:01.474017 IP 211.246.100.49.53 > 10.181.223.215.54007: 26241 3/2/12 A 104.24.123.146, A 172.67.189.102, A 104.24.122.146 (396)
09:00:01.480844 IP 168.126.63.1.53 > 10.181.223.215.54007: 26241 3/2/12 A 172.67.189.102, A 104.24.123.146, A 104.24.122.146 (396)
09:00:01.480848 IP 168.126.63.1.53 > 10.181.223.215.54007: 26431 3/2/12 AAAA 2606:4700:3030::ac43:bd66, AAAA 2606:4700:3034::6818:7b92, AAAA 2606:4700:3030::6818:7a92 (432)
09:00:01.480932 IP 10.181.223.215.45974 > 104.24.123.146.80: Flags [S], seq 21127835, win 64240, options [mss 1460,sackOK,TS val 1542252473 ecr 0,nop,wscale 9], length 0
09:00:01.483772 IP 211.246.100.49.53 > 10.181.223.215.54007: 26431 3/2/12 AAAA 2606:4700:3034::6818:7b92, AAAA 2606:4700:3030::6818:7a92, AAAA 2606:4700:3030::ac43:bd66 (432)
09:00:01.758615 IP 104.24.123.146.80 > 10.181.223.215.45974: Flags [S.], seq 2571105432, ack 21127836, win 65535, options [mss 1400,nop,nop,sackOK,nop,wscale 10], length 0
09:00:01.758703 IP 10.181.223.215.45974 > 104.24.123.146.80: Flags [.], ack 1, win 126, length 0
09:00:01.758758 IP 10.181.223.215.45974 > 104.24.123.146.80: Flags [P.], seq 1:77, ack 1, win 126, length 76: HTTP: GET /ip HTTP/1.1
09:00:02.079214 IP 104.24.123.146.80 > 10.181.223.215.45974: Flags [.], ack 77, win 64, length 0
09:00:02.079248 IP 104.24.123.146.80 > 10.181.223.215.45974: Flags [P.], seq 1:713, ack 77, win 64, length 712: HTTP: HTTP/1.1 200 OK
09:00:02.079269 IP 10.181.223.215.45974 > 104.24.123.146.80: Flags [.], ack 713, win 126, length 0
09:00:02.079279 IP 104.24.123.146.80 > 10.181.223.215.45974: Flags [F.], seq 713, ack 77, win 64, length 0
09:00:02.079358 IP 10.181.223.215.45974 > 104.24.123.146.80: Flags [F.], seq 77, ack 714, win 126, length 0
09:00:02.400265 IP 104.24.123.146.80 > 10.181.223.215.45974: Flags [.], ack 78, win 64, length 0

And from what I mean to do, I modified sockd.conf to make sure Dante uses proxy user:

$ cat /etc/sockd.conf
logoutput: stderr
internal: 127.0.0.1 port = 9898
external: wwp42s0f3u2i12

method: pam none
clientmethod: none

user.notprivileged: proxy

logoutput: stdout

client pass {
        from: 0.0.0.0/0  to: 0.0.0.0/0
}

pass {
        from: 0.0.0.0/0 to: 0.0.0.0/0
        protocol: tcp udp
        log: connect disconnect
}

Finally, I get:

$ curl --proxy 'socks5h://127.0.0.1:9898' https://ifconfig.io/ip
175.223.10.145