By default, containers on one Docker host can’t talk directly to containers on another — each host’s bridge network is its own isolated island. Flannel solves this by creating a flat, cross-host overlay network where every container gets a routable IP that any other host in the cluster can reach directly, no manual port-mapping required. This guide covers what Flannel actually does under the hood, and how to deploy it between plain Docker hosts (not a Kubernetes cluster, though the same technology underpins Kubernetes’ pod networking in many clusters).
What Problem Flannel Solves
Without an overlay network, if container-a on host1 wants to reach container-b on host2, you’d normally need to publish a port on host2 and connect to host2‘s IP — losing the ability to address containers directly and cleanly. Flannel instead gives every host a distinct subnet carved out of a larger cluster-wide CIDR, and every container on that host gets an IP from that subnet. Flannel then handles routing packets between hosts so that any container can reach any other container by IP, cluster-wide, transparently.
How Flannel Works Internally
Flannel runs as a small agent (flanneld) on every host in the cluster. It:
- Registers with a shared key-value store (traditionally etcd, though newer versions also support a Kubernetes API-based backend) to claim a unique subnet for its host from the overall cluster CIDR.
- Configures a local bridge/interface with that subnet.
- Encapsulates packets destined for other hosts’ subnets using one of several backends — most commonly VXLAN, which wraps each packet in a UDP datagram (default port 8472) so it can traverse ordinary IP networks between hosts, even across different L2 segments.
- On the receiving host,
flannelddecapsulates the packet and routes it to the correct local container via the local bridge.
This is conceptually similar to what Docker Swarm’s own overlay networks do internally, but Flannel is independent of Swarm and commonly used with plain Docker hosts or non-Swarm clusters.
Prerequisites
- At least two Linux hosts with Docker installed and network connectivity between them
- A shared etcd cluster reachable from all hosts (Flannel needs somewhere to coordinate subnet allocation)
- Open UDP port 8472 between hosts (VXLAN backend) or your chosen backend’s required port
Step 1: Stand Up etcd
For a lab/demo setup, a single-node etcd instance is enough (production deployments should run a proper 3-node etcd cluster for resilience):
docker run -d --name etcd \
--net host \
quay.io/coreos/etcd:v3.5.15 \
/usr/local/bin/etcd \
--data-dir=/etcd-data --name node1 \
--initial-advertise-peer-urls http://192.168.1.10:2380 \
--listen-peer-urls http://192.168.1.10:2380 \
--advertise-client-urls http://192.168.1.10:2379 \
--listen-client-urls http://192.168.1.10:2379,http://127.0.0.1:2379 \
--initial-cluster node1=http://192.168.1.10:2380 \
--initial-cluster-state new
Verify etcd is healthy:
curl -L http://192.168.1.10:2379/health
{"health":"true"}
Step 2: Define the Cluster-Wide Network Configuration in etcd
Push the overall CIDR Flannel will carve host subnets out of:
curl -L http://192.168.1.10:2379/v2/keys/coreos.com/network/config -XPUT \
-d value='{"Network":"10.244.0.0/16","Backend":{"Type":"vxlan"}}'
Expected response confirms the key was written:
{"action":"set","node":{"key":"/coreos.com/network/config","value":"{\"Network\":\"10.244.0.0/16\",\"Backend\":{\"Type\":\"vxlan\"}}", ...}}
Step 3: Run flanneld on Each Docker Host
On host1 (192.168.1.10) and host2 (192.168.1.11), run the Flannel agent as a privileged container, pointed at the shared etcd:
docker run -d --name flanneld \
--net host \
--privileged \
-v /dev/net:/dev/net \
quay.io/coreos/flannel:v0.25.1 \
/opt/bin/flanneld --etcd-endpoints=http://192.168.1.10:2379 --iface=eth0
Check the logs to confirm it acquired a subnet lease:
docker logs flanneld
I0729 subnet.go:207] Subnet lease acquired: 10.244.15.0/24
I0729 vxlan.go:137] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
Each host gets its own /24 slice out of the /16 you defined — host1 might get 10.244.15.0/24 and host2 might get 10.244.22.0/24.
Flannel writes its allocated subnet info to /run/flannel/subnet.env on each host:
cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.15.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
Step 4: Configure Docker to Use Flannel’s Subnet
By default, Docker’s own bridge network doesn’t know about Flannel’s allocation. Point Docker’s bridge at the subnet Flannel assigned to this specific host, by adding it to daemon.json:
source /run/flannel/subnet.env
sudo tee /etc/docker/daemon.json <<EOF
{
"bip": "${FLANNEL_SUBNET}",
"mtu": ${FLANNEL_MTU}
}
EOF
sudo systemctl restart docker
Confirm Docker’s bridge now uses the Flannel-assigned range:
ip addr show docker0
docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450
inet 10.244.15.1/24 brd 10.244.15.255 scope global docker0
Repeat Steps 3 and 4 on every host, each picking up its own unique subnet automatically.
Step 5: Test Cross-Host Container Connectivity
On host1, start a container:
docker run -d --name test-a alpine sleep 3600
docker exec test-a ip addr show eth0
inet 10.244.15.2/24 brd 10.244.15.255 scope global eth0
On host2, start another and ping across hosts directly by container IP:
docker run -d --name test-b alpine sleep 3600
docker exec test-b ping -c 3 10.244.15.2
PING 10.244.15.2 (10.244.15.2): 56 data bytes
64 bytes from 10.244.15.2: seq=0 ttl=62 time=0.412 ms
64 bytes from 10.244.15.2: seq=1 ttl=62 time=0.387 ms
64 bytes from 10.244.15.2: seq=2 ttl=62 time=0.401 ms
--- 10.244.15.2 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
If this succeeds, you have working cross-host container networking with no manual port publishing involved — Flannel routed the packet from host2‘s bridge, VXLAN-encapsulated it, sent it to host1, where it was decapsulated and delivered straight to test-a‘s network namespace.
Verifying the VXLAN Path
Inspect the VXLAN interface Flannel created:
ip -d link show flannel.1
flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450
vxlan id 1 local 192.168.1.10 dev eth0 srcport 0 0 dstport 8472
Capture traffic to confirm encapsulation is actually happening (run on the underlying host interface, not inside a container):
sudo tcpdump -i eth0 udp port 8472 -c 5
Internal Working: Why the MTU Matters
VXLAN adds roughly 50 bytes of overhead per packet (outer IP + UDP + VXLAN headers). If your physical network’s MTU is the standard 1500 and you don’t account for this overhead, packets get fragmented or dropped, causing mysterious connectivity issues that look like networking is broken when it’s actually an MTU mismatch. This is exactly why Flannel writes FLANNEL_MTU=1450 into subnet.env and why the Docker bridge configuration above explicitly sets mtu to match — every container interface downstream needs to respect that reduced MTU.
Choosing a Backend
VXLAN is the most portable and commonly used backend since it only needs standard UDP connectivity between hosts. Other backends exist for specific scenarios:
- host-gw — no encapsulation at all; relies on direct L2 connectivity and manipulates host routing tables directly. Faster (no encapsulation overhead) but only works when all hosts are on the same L2 network segment.
- udp — a slower, userspace fallback, mostly historical at this point; VXLAN largely superseded it.
Switch backends by changing the Backend.Type value in the etcd network config from Step 2.
Security Considerations
- VXLAN traffic is unencrypted by default — anyone who can sniff traffic between hosts can read container-to-container communication. For sensitive workloads, run Flannel over a WireGuard or IPsec tunnel between hosts, or use Flannel’s experimental IPsec backend if your version supports it.
- Restrict UDP 8472 (or your chosen backend’s port) to only the hosts in your cluster via firewall rules — this port shouldn’t be reachable from outside your trusted network.
- Secure etcd itself with client certificate authentication in production; an open etcd endpoint lets anyone rewrite your cluster’s network configuration.
sudo ufw allow from 192.168.1.0/24 to any port 8472 proto udp
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
flanneld fails to acquire a lease | Can’t reach etcd | Verify etcd is running and reachable: curl -L http://etcd-host:2379/health |
| Containers on different hosts can’t ping each other | UDP 8472 blocked between hosts | Open the VXLAN port in firewall rules on all hosts |
| Intermittent connectivity, larger payloads fail | MTU mismatch between Docker bridge and Flannel’s advertised MTU | Ensure daemon.json‘s mtu matches FLANNEL_MTU from subnet.env |
| Two hosts get overlapping subnets | Stale etcd data from a previous cluster config | Clear the /coreos.com/network etcd prefix and redeploy from Step 2 |
Summary
Flannel gives Docker hosts a flat, cluster-wide network where every container gets a directly routable IP, using etcd for coordination and VXLAN (by default) to tunnel traffic between hosts. Once configured, container-to-container communication across hosts works exactly like it would within a single host’s bridge network — no manual port mapping, no NAT gymnastics. Pay close attention to MTU settings and firewall rules for the VXLAN port, since these are the two most common sources of subtle connectivity problems.
References
- Flannel project documentation: https://github.com/flannel-io/flannel
- Flannel backend types: https://github.com/flannel-io/flannel/blob/master/Documentation/backends.md
- etcd documentation: https://etcd.io/docs/
- CNCF networking landscape: https://landscape.cncf.io/
