pathvector-dev

Originally published at https://blog.pathvector.dev/protocol-lab-ecmp-32/ — part of the free Protocol Lab series.

This post is part of Protocol Lab, a free, hands-on series for learning networking protocols by building and breaking them in a container lab. All the lab material — topologies, configs, and scripts — lives in the repo: github.com/pathvector-studio/protocol-lab.

In Lab #31, anycast had routing pick one best path out of many. ECMP (equal-cost multipath) does the opposite trick: when several paths tie, routing keeps all of them and spreads traffic across them — hashing each flow onto one link.

Reading guide: rfc-notes/ecmp.md

Prerequisite: Lab 31: Anycast — One Address, Many Servers, Routing Decides

Expected time: 40–55 minutes.

The Goal

Two routers are joined by two parallel links. r1 learns the server subnet over both eBGP sessions and, with maximum-paths 2, installs a two-next-hop route. When the client opens many TCP flows to the server:

  • with L4 hashing (fib_multipath_hash_policy=1), the flows split roughly evenly across both links,
  • with the default L3 hashing (=0), every flow shares the same src/dst IP and so hashes to the same link — the other sits idle. That's the classic ECMP gotcha.

By the end, you should be able to explain this table:

hash policy what's hashed 16 flows, same client→server
0 (L3, default) src/dst IP all on one link (other ~0)
1 (L4) IP + ports ~half on each link

What You Will Learn

  • What ECMP is and how maximum-paths puts several equal paths in the FIB.
  • Why routers hash per flow (5-tuple), not per packet (reordering).
  • How Linux's fib_multipath_hash_policy decides L3 vs L4 hashing.
  • The gotcha: same src/dst IP + L3 hashing → one link does all the work.
  • How ECMP differs from anycast (Lab 31): keep many paths vs pick one.

This lab does not cover:

  • Weighted / unequal-cost multipath (UCMP).
  • LAG / bonding (L2 link aggregation) — similar hashing, different layer.
  • Per-packet spraying and flowlet switching.

Where to Read in the RFCs

Reference What to focus on
RFC 2992 The per-flow (hash-threshold) approach
RFC 4271 §9.1 + maximum-paths The conditions for multiple equal-cost paths to land in the RIB
RFC 7424 Uneven distribution in real traffic (flow entropy)
RFC 5737 / RFC 1918 Confirming the addresses used here are local/documentation-only

The Big Picture

Behind the client sits r1; behind r2 sits the server. r1 and r2 connect over eBGP across two parallel links.

 client            r1 (AS 65001)      link A: 10.0.12.0/30      r2 (AS 65002)         server
 10.0.9.2 -- eth1 --+-- eth2 =========================== eth1 --+-- eth3 -- 10.0.8.2
                    +-- eth3 =========================== eth2 --+
                            link B: 10.0.13.0/30

Enter fullscreen mode Exit fullscreen mode

r1 learns 10.0.8.0/24 over both links and, thanks to maximum-paths 2, installs a two-next-hop ECMP route in its FIB. Many flows then hash across both links.

flowchart LR
  C["client<br/>iperf3 -c -P 16"] --> R1["r1<br/>ECMP route:<br/>via eth2 + via eth3"]
  R1 -->|"flows hashed<br/>(L4: by port)"| LA["link A (eth2)"]
  R1 -->|"other flows"| LB["link B (eth3)"]
  LA --> R2["r2"]
  LB --> R2
  R2 --> S["server<br/>iperf3 -s"]

Enter fullscreen mode Exit fullscreen mode

10.0.0.0/8 is a local, closed range.

Note: Everything here uses local/documentation address space (RFC 1918 / RFC 5737), so nothing in this lab touches the real internet.

What You Need

Recommended environment:

  • Linux / WSL2 / a Linux VM
  • Docker
  • containerlab

Images used:

  • frrouting/frr:latest — BGP multipath on r1 and r2.
  • nicolaka/netshoot:latest — the client and the server, providing iperf3.

No additional images are required.

Running the Lab

The quick path, which deploys, verifies, and tears down for you:

./scripts/labctl.sh run ecmp-32

Enter fullscreen mode Exit fullscreen mode

Or step through it manually:

1. Move into the working directory 

cd protocol-lab/examples/ecmp-32

Enter fullscreen mode Exit fullscreen mode

2. Deploy 

sudo containerlab deploy -t ecmp-32.clab.yml

Enter fullscreen mode Exit fullscreen mode

r1 and r2 come up with maximum-paths 2 and fib_multipath_hash_policy=1 (L4 hashing).

3. Confirm the ECMP route 

docker exec clab-ecmp-32-r1 ip route show 10.0.8.0/24

Enter fullscreen mode Exit fullscreen mode

You should see two nexthop entries (via 10.0.12.2 and via 10.0.13.2).

4. Push many flows and watch both links 

docker exec -d clab-ecmp-32-server iperf3 -s
# before
docker exec clab-ecmp-32-r1 cat /sys/class/net/eth2/statistics/tx_bytes
docker exec clab-ecmp-32-r1 cat /sys/class/net/eth3/statistics/tx_bytes
# 16 parallel flows
docker exec clab-ecmp-32-client iperf3 -c 10.0.8.2 -P 16 -t 6
# after — both counters moved
docker exec clab-ecmp-32-r1 cat /sys/class/net/eth2/statistics/tx_bytes
docker exec clab-ecmp-32-r1 cat /sys/class/net/eth3/statistics/tx_bytes

Enter fullscreen mode Exit fullscreen mode

The tx_bytes counters on both links grow roughly half-and-half.

5. See the gotcha: switch back to L3 hashing 

docker exec clab-ecmp-32-r1 sysctl -w net.ipv4.fib_multipath_hash_policy=0
docker exec clab-ecmp-32-client iperf3 -c 10.0.8.2 -P 16 -t 5
# only one link's counter grows (every flow shares the same src/dst IP → same hash)
docker exec clab-ecmp-32-r1 sysctl -w net.ipv4.fib_multipath_hash_policy=1   # restore

Enter fullscreen mode Exit fullscreen mode

Expected Output

  • ip route show 10.0.8.0/24: two nexthops.
  • policy=1: with 16 flows, eth2/eth3 tx_bytes grow roughly half-and-half (about 131 GB / 133 GB in this environment).
  • policy=0: traffic piles onto one link (one carries nearly everything, the other stays near 0).

Why It Works

ECMP (equal-cost multipath) means "when several equally good paths exist, use them all."

  • Where the multiple paths come from. By default, BGP installs exactly one best route per prefix (Lab 31). With maximum-paths 2, up to two tied paths go into the FIB. Here, r1–r2 run eBGP over two parallel links, and both sessions advertise the same prefix with the same AS_PATH length — a tie, so r1 installs a two-next-hop ECMP route.
  • Per-flow hashing. A router doesn't alternate packets between the links. If one connection's packets took different paths, latency differences would cause reordering, which TCP can mistake for loss. Instead, the router hashes each packet's 5-tuple (src/dst IP, protocol, src/dst port) and pins the same flow to the same next-hop forever. Different flows scatter independently.
  • What goes into the hash. Linux's fib_multipath_hash_policy decides: 0 = L3 (IPs only), 1 = L4 (IPs + ports). This lab has one client talking to one server, so every flow shares the same src/dst IPs. With L3 hashing, every flow gets the same hash value and piles onto one link. With L4 hashing, the ports are included, so each flow — with its distinct source port — hashes independently and both links get used.
  • How even is "even"? More flows means the split approaches statistically even; few flows means skew. In this lab, 16 flows land roughly half-and-half. A single flow is capped at one link — what ECMP raises is the aggregate of many flows.

The key insight: routing keeps multiple equal paths, and the kernel hashes each flow onto one of them. It's the dual of anycast (which narrows to one path) — another face of the same BGP path selection.

Common Pitfalls

  • Assuming ECMP alternates packets. It hashes per flow, precisely to avoid reordering.
  • Expecting both links to be used with one IP pair. Under L3 hashing (the default), everything lands on one link. You need L4 hashing (policy=1). This is the biggest trap.
  • Expecting a perfectly even split. With few flows, the split skews. Evenness is a statistical property of many flows.
  • Expecting a single connection to get faster. One flow stays on one link. ECMP raises total throughput, not per-flow throughput.
  • Forgetting maximum-paths. Without it, only one best path is installed and you get no ECMP.
  • Reading too much into veth speed. Virtual links are extremely fast; the absolute numbers in this lab (hundreds of Gbit/s) are environment-dependent. What matters is the ratio between the two links.

Cleanup 

sudo containerlab destroy -t ecmp-32.clab.yml --cleanup

Enter fullscreen mode Exit fullscreen mode

If you used labctl.sh run ecmp-32, the script runs destroy for you at the end.

Check Your Understanding

  1. What is ECMP? How does BGP end up installing multiple equal-cost routes in the FIB?
  2. Why do routers spread traffic per flow rather than per packet? What would per-packet spreading cause?
  3. What's the difference between fib_multipath_hash_policy 0 and 1?
  4. With one client talking to one server, why does L3 hashing concentrate everything on a single link?
  5. Contrast ECMP with anycast (Lab 31) in terms of number of paths kept and purpose.
  6. Does ECMP raise the throughput of a single TCP connection? Why or why not?

References

Verified Run Log (2026-07-07)

This lab has been confirmed reproducible on real hardware.

Environment:

  • Ubuntu 26.04 LTS (kernel 7.0.0-27-generic, x86_64)
  • Docker 29.1.3
  • containerlab 0.77.0
  • r1 / r2: frrouting/frr:latest (BGP multipath)
  • client / server: nicolaka/netshoot:latest (iperf3)

Running PATH="/tmp/pl-shim:$PATH" ./scripts/labctl.sh run ecmp-32 performed deploy → verify → destroy, and verification.json returned "status": "verified".

The two-next-hop ECMP route 

10.0.8.0/24 nhid 25 proto bgp metric 20
    nexthop via 10.0.12.2 dev eth2 weight 1
    nexthop via 10.0.13.2 dev eth3 weight 1

Enter fullscreen mode Exit fullscreen mode

Thanks to maximum-paths 2, r1 installed two equal-cost paths to the server subnet in its FIB as a next-hop group.

L4 hashing splits both links almost evenly

With 16 parallel TCP flows (iperf3 -c 10.0.8.2 -P 16), measuring tx_bytes on r1's two egress links:

policy: 1
eth2 (link A) tx delta: 131801067437 bytes   (49.6%)
eth3 (link B) tx delta: 133902251068 bytes   (50.4%)
total:                  265703318505 bytes

Enter fullscreen mode Exit fullscreen mode

Under fib_multipath_hash_policy=1 (L4, ports included), each flow — with its distinct source port — scattered almost evenly across both links.

The gotcha: L3 hashing piles everything onto one link

Running the same 16 flows under fib_multipath_hash_policy=0 (L3, IPs only):

L3 hashing (policy=0):
  eth2 delta: 151 bytes
  eth3 delta: 133854511558 bytes

Enter fullscreen mode Exit fullscreen mode

Every flow shares the same src/dst IPs (one client → one server), so the L3 hash is identical for all of them, and nearly everything landed on a single link (eth2's 151 bytes is effectively zero). To make ECMP actually work here, you need the L4 policy that includes ports in the hash — this is the trap people fall into most.

Cleanup 

containerlab destroy -t ecmp-32.clab.yml --cleanup

Enter fullscreen mode Exit fullscreen mode

That's ECMP: routing keeps every path that ties, and the kernel hashes each flow onto one of them — just make sure the ports are in the hash, or one link will quietly do all the work.

Explore the full Protocol Lab series here: github.com/pathvector-studio/protocol-lab. If these labs are useful to you, please ⭐ star the repo on GitHub — it genuinely helps others find the project.

Next up, we'll keep pulling on the multipath thread — think weighted paths and what happens when links aren't equal after all.