Injecting latency into applications running in docker
When building decoupled systems that communicate over the network it is valuable to test them under less than ideal network situations1. In this post we’ll cover a very simple example: two docker hosts communicating over the network, with platform enforced latency between them.
tc
The traffic control utility, tc(8), will be our workhorse here. It’s been around since at least 2001 and can manipulate network interfaces, adjusting bandwidth, latency, bustable latency and more.
docker network
For this example we create a docker network for the contianers. It isn’t strctly necessary but we can hit the other containers by dns name if we do it.
$ docker network create pingnet
0232f33634480a23fc62621f3810591491623d9ee064bf1ff8ca678185c37ac3
docker
We only need two simple Docker containers. Get two shells up and run:
# shell 1
$ docker run --cap-add=NET_ADMIN --network=pingnet --name ping1 -it -rm quay.io/nibalizer/utilities /bin/bash
# shell 2
$ docker run --network=pingnet -it --rm --name ping2 quay.io/nibalizer/utilities /bin/bash
Now test pinging ping2 from ping 1.
root@2d6aab9222dd:/# ping -c 5 ping2
PING ping2 (172.19.0.3) 56(84) bytes of data.
64 bytes from ping2.pingnet (172.19.0.3): icmp_seq=1 ttl=64 time=0.297 ms
64 bytes from ping2.pingnet (172.19.0.3): icmp_seq=2 ttl=64 time=0.177 ms
64 bytes from ping2.pingnet (172.19.0.3): icmp_seq=3 ttl=64 time=0.178 ms
64 bytes from ping2.pingnet (172.19.0.3): icmp_seq=4 ttl=64 time=0.176 ms
64 bytes from ping2.pingnet (172.19.0.3): icmp_seq=5 ttl=64 time=0.175 ms
--- ping2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4080ms
rtt min/avg/max/mdev = 0.175/0.200/0.297/0.048 ms
Cool ping works and there is a baseline of about 0.175ms between hosts.
add latency
In the ping1 container host (shell 1). Run the following tc command. This will set latency to 270ms. This can only be run in ping1 becuase that’s where we’ve added --cap-add=NET_ADMIN.
# tc qdisc add dev eth0 root netem delay 270ms
Now test latency via ping:
root@2d6aab9222dd:/# ping -c 5 ping2
PING ping2 (172.19.0.3) 56(84) bytes of data.
64 bytes from ping2.pingnet (172.19.0.3): icmp_seq=1 ttl=64 time=270 ms
64 bytes from ping2.pingnet (172.19.0.3): icmp_seq=2 ttl=64 time=270 ms
64 bytes from ping2.pingnet (172.19.0.3): icmp_seq=3 ttl=64 time=270 ms
64 bytes from ping2.pingnet (172.19.0.3): icmp_seq=4 ttl=64 time=270 ms
64 bytes from ping2.pingnet (172.19.0.3): icmp_seq=5 ttl=64 time=270 ms
270ms latency is observed!
You can clear the tc rules via:
# tc qdisc del dev eth0 root
TC has lots more options, and it’s possible to expand this strategy to more containers and more rules. Check out the tc manpage and tylertreat/comcast to go further.
Special thanks to Caleb Boylan and Greg Haynes for their help figuring this out and reviewing the post.
-
At scale, this is sometimes called Chaos Engineering. There are toolsets and companies that can help you get started with “Chaos Engineering” if you’d like. ↩︎
