Before overlay network drivers like Weave or Docker’s built-in overlay driver existed, the way I learned cross-host container networking was the hard way: building a GRE (Generic Routing Encapsulation) tunnel by hand between two Linux hosts and bridging it into Docker’s networking stack. I still think every container networking engineer should do this once, because it strips away all the abstraction and shows you exactly what an “overlay network” is actually doing at the packet level. This guide walks through building that tunnel manually.
What GRE Actually Is
GRE is a tunneling protocol, originally designed by Cisco and standardized in RFC 2784, that encapsulates one IP packet inside another. It doesn’t provide encryption — it’s purely a way to make two networks that aren’t directly routable to each other behave as if they’re on the same link. A GRE tunnel between two hosts creates a virtual point-to-point interface on each side; anything routed into that interface gets wrapped in a GRE header and an outer IP header, sent across the real network, and unwrapped on the other end.
This is exactly the mechanism that lets Docker bridges on two separate hosts be joined into a single logical Layer 2 network: I connect each host’s Docker bridge to a local end of the GRE tunnel, and traffic between containers on different hosts flows through the tunnel as if the two bridges were one.
Prerequisites
- Two Linux hosts (bare metal, VMs, or EC2 instances) with root/sudo access.
- Docker installed on both.
- IP forwarding capability and the
ipcommand (iproute2package). - Direct IP reachability between the two hosts (public IP, VPN, or same VPC/subnet). I’ll use example IPs
192.168.1.10(host A) and192.168.1.20(host B).
Step 1: Inspect the Existing Docker Bridge
By default Docker creates docker0:
ip addr show docker0
Typical output:
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
The problem: both hosts use the same 172.17.0.0/16 range by default, which would collide if bridged directly. So the first real step is giving each host’s Docker bridge a distinct subnet.
Step 2: Assign Non-Overlapping Subnets
On host A, edit or create /etc/docker/daemon.json:
{
"bip": "172.18.1.1/24"
}
On host B:
{
"bip": "172.18.2.1/24"
}
Restart Docker on both:
sudo systemctl restart docker
Confirm the new bridge subnet:
ip addr show docker0
Host A should now show 172.18.1.1/24, host B 172.18.2.1/24.
Step 3: Create the GRE Tunnel Interface
On host A (local IP 192.168.1.10, remote IP 192.168.1.20):
sudo ip tunnel add gre1 mode gre remote 192.168.1.20 local 192.168.1.10 ttl 255
sudo ip link set gre1 up
sudo ip addr add 10.100.0.1/30 dev gre1
On host B (local IP 192.168.1.20, remote IP 192.168.1.10):
sudo ip tunnel add gre1 mode gre remote 192.168.1.10 local 192.168.1.20 ttl 255
sudo ip link set gre1 up
sudo ip addr add 10.100.0.2/30 dev gre1
Test basic tunnel connectivity before touching Docker at all:
# from host A
ping -c 3 10.100.0.2
Expected output:
PING 10.100.0.2 (10.100.0.2) 56(84) bytes of data.
64 bytes from 10.100.0.2: icmp_seq=1 ttl=64 time=0.612 ms
64 bytes from 10.100.0.2: icmp_seq=2 ttl=64 time=0.487 ms
64 bytes from 10.100.0.2: icmp_seq=3 ttl=64 time=0.501 ms
If this doesn’t work, stop here and check security groups / firewall rules for GRE (IP protocol 47) before going further — GRE isn’t TCP or UDP, so many cloud security groups and iptables rulesets silently drop it unless you explicitly allow protocol 47.
Step 4: Route Container Subnets Through the Tunnel
Now I tell each host how to reach the other host’s Docker subnet via the tunnel.
On host A:
sudo ip route add 172.18.2.0/24 via 10.100.0.2 dev gre1
On host B:
sudo ip route add 172.18.1.0/24 via 10.100.0.1 dev gre1
Step 5: Allow Forwarding and Fix Docker’s iptables Isolation
Docker installs iptables rules by default that block forwarding between the bridge and other interfaces unless explicitly allowed. Enable IP forwarding and open the path:
sudo sysctl -w net.ipv4.ip_forward=1
sudo iptables -A FORWARD -i docker0 -o gre1 -j ACCEPT
sudo iptables -A FORWARD -i gre1 -o docker0 -j ACCEPT
Run the equivalent on both hosts.
Step 6: Test Container-to-Container Connectivity
On host A:
docker run -d --name c1 busybox sleep 3600
docker exec c1 ip addr show eth0
Output shows something like:
inet 172.18.1.2/24 brd 172.18.1.255 scope global eth0
On host B:
docker run -d --name c2 busybox sleep 3600
docker exec c2 ip addr show eth0
Output:
inet 172.18.2.2/24 brd 172.18.2.255 scope global eth0
Now ping across hosts, container to container, through the GRE tunnel:
docker exec c1 ping -c 3 172.18.2.2
Expected:
PING 172.18.2.2 (172.18.2.2): 56 data bytes
64 bytes from 172.18.2.2: seq=0 ttl=63 time=1.402 ms
64 bytes from 172.18.2.2: seq=1 ttl=63 time=1.187 ms
64 bytes from 172.18.2.2: seq=2 ttl=63 time=1.203 ms
That confirms two containers on two completely separate Docker hosts are exchanging packets over a manually built GRE tunnel, with no third-party overlay tool involved at all.
Persisting the Configuration
Manual ip commands disappear on reboot. I persist them with a systemd unit rather than relying on distro-specific network scripts:
# /etc/systemd/system/gre-tunnel.service
[Unit]
Description=GRE tunnel to remote Docker host
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/ip tunnel add gre1 mode gre remote 192.168.1.20 local 192.168.1.10 ttl 255
ExecStart=/usr/sbin/ip link set gre1 up
ExecStart=/usr/sbin/ip addr add 10.100.0.1/30 dev gre1
ExecStart=/usr/sbin/ip route add 172.18.2.0/24 via 10.100.0.2 dev gre1
ExecStop=/usr/sbin/ip tunnel del gre1
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now gre-tunnel.service
Security Considerations
GRE has zero built-in encryption or authentication — anyone who can see the packets on the wire between the two hosts can read and potentially spoof traffic. On AWS, if both hosts are inside the same VPC, this is generally acceptable since the underlying network is already isolated, but if the tunnel crosses the public internet I strongly recommend wrapping it in IPsec (GRE-over-IPsec) rather than running bare GRE:
# Example: strongSwan IPsec transport-mode config protecting GRE
# /etc/ipsec.conf
conn gre-protect
left=192.168.1.10
right=192.168.1.20
type=transport
authby=secret
ike=aes256-sha2_256-modp2048
esp=aes256-sha2_256
auto=start
Troubleshooting
- No connectivity through the tunnel but
pingto tunnel endpoints works: checkiptables -L FORWARD -vfor a rule silently dropping traffic betweendocker0andgre1. - GRE traffic never arrives: verify AWS/cloud security groups explicitly allow IP protocol 47 (GRE isn’t TCP/UDP, so numbered protocol rules are required, not port rules).
- MTU-related packet loss for large payloads: GRE encapsulation adds overhead (24 bytes), so set the tunnel interface MTU explicitly:
ip link set gre1 mtu 1476.
Best Practices
- Treat manual GRE tunnels as a learning tool or a fallback for environments where you can’t run a full overlay daemon — for production fleets, prefer Docker’s native
overlaydriver (with Swarm mode) or a CNI-based solution, since they handle key exchange, multiple peers, and failure recovery automatically. - Always test raw tunnel connectivity (
pingbetween tunnel endpoint IPs) before adding Docker into the picture, so you know whether a problem is in routing or in Docker’s iptables rules. - Document and automate tunnel setup with systemd or configuration management (Ansible/Terraform) rather than leaving it as tribal knowledge on a single host.
Multiple Tunnels and Scaling Beyond Two Hosts
Everything above assumes exactly two hosts, which is where GRE is easiest to reason about — it’s inherently a point-to-point technology. The moment I need a third host, I have to decide between two approaches, and both have real tradeoffs worth understanding before choosing one.
Full mesh of point-to-point tunnels. Each host gets a separate GRE interface and route for every other host in the cluster. With three hosts that’s three tunnels; with ten hosts it’s forty-five. This scales quadratically, so I’ve only ever used it for small, mostly-static clusters where I know the host count isn’t going to grow much.
# On host A, in addition to the tunnel to host B, add one to host C
sudo ip tunnel add gre2 mode gre remote 192.168.1.30 local 192.168.1.10 ttl 255
sudo ip link set gre2 up
sudo ip addr add 10.100.1.1/30 dev gre2
sudo ip route add 172.18.3.0/24 via 10.100.1.2 dev gre2
A hub-and-spoke design. One host acts as a central relay, and every other host tunnels only to it, trading a single point of failure for far less configuration per new host added. This is closer to what dedicated overlay tools like Weave do automatically with their gossip-based mesh, which is exactly why, past two or three hosts, I generally recommend reaching for a real overlay driver rather than continuing to hand-manage GRE routes.
Comparing GRE to VXLAN
GRE and VXLAN solve a similar problem but differ in one detail that matters a lot in practice: GRE encapsulates over raw IP (protocol 47), while VXLAN encapsulates over UDP (typically port 4789). Because UDP is far more commonly allowed through load balancers, NAT gateways, and cloud security groups than a raw IP protocol number, I’ve had noticeably fewer connectivity surprises building tunnels with VXLAN in cloud environments — something worth keeping in mind if the GRE approach in this guide runs into unexplained packet loss on infrastructure you don’t fully control.
Summary
Building a GRE tunnel between two Docker hosts by hand demystifies what overlay networking tools are actually doing under the hood: encapsulating packets, exchanging routes, and bridging otherwise-isolated container subnets into a single logical network. It’s not something I’d run in production at scale — Docker’s built-in overlay driver or a CNI plugin does this more robustly — but understanding it makes every higher-level tool (Weave, Flannel, Calico) far easier to reason about when something breaks.
References
- RFC 2784: Generic Routing Encapsulation: https://www.rfc-editor.org/rfc/rfc2784
- Docker network drivers overview: https://docs.docker.com/network/drivers/
- Docker overlay network driver: https://docs.docker.com/network/drivers/overlay/
- Linux
ip-tunnelman page reference via kernel.org networking docs: https://www.kernel.org/doc/html/latest/networking/
