Spencer Krum

When building decoupled systems that communicate over the network it is valuable to test them under less than ideal network situations¹. In this post we’ll cover a very simple example: two docker hosts communicating over the network, with platform enforced latency between them.

tc

The traffic control utility, tc(8), will be our workhorse here. It’s been around since at least 2001 and can manipulate network interfaces, adjusting bandwidth, latency, bustable latency and more.

docker network

For this example we create a docker network for the contianers. It isn’t strctly necessary but we can hit the other containers by dns name if we do it.

$ docker network create pingnet
0232f33634480a23fc62621f3810591491623d9ee064bf1ff8ca678185c37ac3

docker

We only need two simple Docker containers. Get two shells up and run:

# shell 1
$ docker run --cap-add=NET_ADMIN --network=pingnet --name ping1 -it -rm quay.io/nibalizer/utilities /bin/bash

# shell 2
$ docker run --network=pingnet -it --rm --name ping2 quay.io/nibalizer/utilities /bin/bash

Now test pinging ping2 from ping 1.

root@2d6aab9222dd:/# ping -c 5 ping2
PING ping2 (172.19.0.3) 56(84) bytes of data.
64 bytes from ping2.pingnet (172.19.0.3): icmp_seq=1 ttl=64 time=0.297 ms
64 bytes from ping2.pingnet (172.19.0.3): icmp_seq=2 ttl=64 time=0.177 ms
64 bytes from ping2.pingnet (172.19.0.3): icmp_seq=3 ttl=64 time=0.178 ms
64 bytes from ping2.pingnet (172.19.0.3): icmp_seq=4 ttl=64 time=0.176 ms
64 bytes from ping2.pingnet (172.19.0.3): icmp_seq=5 ttl=64 time=0.175 ms

--- ping2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4080ms
rtt min/avg/max/mdev = 0.175/0.200/0.297/0.048 ms

Cool ping works and there is a baseline of about 0.175ms between hosts.

add latency

In the ping1 container host (shell 1). Run the following tc command. This will set latency to 270ms. This can only be run in ping1 becuase that’s where we’ve added --cap-add=NET_ADMIN.

# tc qdisc add dev eth0 root netem delay 270ms

Now test latency via ping:

root@2d6aab9222dd:/# ping -c 5 ping2
PING ping2 (172.19.0.3) 56(84) bytes of data.
64 bytes from ping2.pingnet (172.19.0.3): icmp_seq=1 ttl=64 time=270 ms
64 bytes from ping2.pingnet (172.19.0.3): icmp_seq=2 ttl=64 time=270 ms
64 bytes from ping2.pingnet (172.19.0.3): icmp_seq=3 ttl=64 time=270 ms
64 bytes from ping2.pingnet (172.19.0.3): icmp_seq=4 ttl=64 time=270 ms
64 bytes from ping2.pingnet (172.19.0.3): icmp_seq=5 ttl=64 time=270 ms

270ms latency is observed!

You can clear the tc rules via:

# tc qdisc del dev eth0 root

TC has lots more options, and it’s possible to expand this strategy to more containers and more rules. Check out the tc manpage and tylertreat/comcast to go further.

Special thanks to Caleb Boylan and Greg Haynes for their help figuring this out and reviewing the post.