Skip to content

PoC: ZeroTier pair (historical)

Historical

This first PoC used ZeroTier. The data plane is now WireGuard + VXLAN (see the decision log and the WireGuard PoC). The findings about OpenWrt below still apply.

Date: 2026-10-25. OpenWrt 25.12.6 armsr/armv8, QEMU with HVF on Apple Silicon (native arm64), ZeroTier 1.16.2 on the routers, a self-hosted ZeroTier controller (1.14.2) in Docker standing in for ZeroTier Central.

Result

A laptop VM plugged into the travel router's guest port got its address from the home router's ISP box by DHCP, reached the home router and the internet through the home connection. Bridge tables prove the path: the home router learned the laptop's MAC on its ZeroTier port, the travel router on its guest port.

flowchart LR
  laptop["laptop VM<br/>10.0.2.16 (from home DHCP)"] -- eth1 --- tg["travel eth2"]
  subgraph travel["travel router"]
    tg --- bg["br-guest (no IP)"] --- tzt["ztx2jtovre"]
  end
  tzt <-- "ZeroTier L2 (activeBridge)" --> hzt
  subgraph home["home router"]
    hzt["ztx2jtovre"] --- bi["br-isp (DHCP 10.0.2.15)"] --- he["eth1"]
  end
  he --- isp["ISP box (QEMU user net, DHCP + NAT)"]

Also verified on the same VMs:

  • A Jumo image (agent, procd service, fleet code, control plane URL in the read-only root) enrolls on first boot with no manual step.
  • firstboot -y && reboot (factory reset) wipes the overlay, keeps the enroll code, and the agent re-enrolls by itself with the same device id.
  • ea status / ea doctor on the box; agent RSS 8.4 MB.

Findings that shape the agent

  1. Ship /etc/rc.d/S99edge-agent in the image. Enabling the service from uci-defaults only takes effect on the next boot: OpenWrt lists /etc/rc.d before uci-defaults run. With the link in the read-only root, the agent starts on first boot and after every factory reset.
  2. Restart zerotier-one after moving the uplink into a bridge. On the home router, putting eth1 into br-isp left ZeroTier without a working UDP path (fell back to a TCP relay, then stuck in REQUESTING_CONFIGURATION). A restart fixed it. The applier must order: bridge, then ZeroTier restart.
  3. Re-add the ZeroTier port to the bridge after a ZeroTier restart. The zt* interface is recreated; it was not back in br-isp until added. The applier must check membership after every ZeroTier (re)start.
  4. The ZeroTier node id changes after a factory reset (the package generates the secret on first start, stored in the overlay). The agent reports the node id; the control plane must authorize the new one and deauthorize the old one.
  5. Both routers must be activeBridge: each carries frames from MACs that are not its own.
  6. The ZeroTier interface name is deterministic per network (ztx2jtovre for a9d6a01745974161, same on both routers), so the applier can compute it, or read portDeviceName from zerotier-cli -j listnetworks.
  7. Never replace a running binary in place (ETXTBSY): copy to a temp name and rename, as digilan's updater does.

Open: direct path vs relay

Here both routers sit behind NAT on the same Mac and reach each other only through ZeroTier's relays (150–300 ms). In the field (home and hotel on different public IPs) ZeroTier normally punches a direct UDP path. This must be measured on real hardware early: relayed traffic would make the product slow. Next step: one router at home, one on a phone hotspot.

Reproduce

cd firmware
scripts/build-jumo-image.sh --enroll-code <fleet code> \
  --control-plane http://10.0.2.2:8787 --packages zerotier       # ImageBuilder, ~2.5 min
gunzip -c build/armsr-armv8-generic/images/*squashfs-combined-efi.img.gz > build/vm/jumo-ib.img
JUMO_IMAGE=build/vm/jumo-ib.img scripts/poc-vm.sh start home
JUMO_IMAGE=build/vm/jumo-ib.img scripts/poc-vm.sh start travel
scripts/poc-vm.sh start laptop

Then, on each router (this is what the agent's ZeroTier applier will do):

uci set zerotier.global.enabled=1; uci -q delete zerotier.earth
uci set zerotier.jumo=network; uci set zerotier.jumo.id=<network id>
uci set zerotier.jumo.allow_managed=0; uci commit zerotier; /etc/init.d/zerotier restart

# home: uplink + ZeroTier in one bridge, router keeps DHCP on it
uci set network.br_isp=device; uci set network.br_isp.type=bridge; uci set network.br_isp.name=br-isp
uci add_list network.br_isp.ports=eth1; uci add_list network.br_isp.ports=<zt dev>
uci set network.wan.device=br-isp; uci set network.wan6.device=br-isp

# travel: guest port + ZeroTier in one bridge, no IP
uci set network.br_guest=device; uci set network.br_guest.type=bridge; uci set network.br_guest.name=br-guest
uci add_list network.br_guest.ports=eth2; uci add_list network.br_guest.ports=<zt dev>
uci set network.guest=interface; uci set network.guest.proto=none; uci set network.guest.device=br-guest
uci commit network; /etc/init.d/network reload

Authorize both node ids on the controller with {"authorized": true, "activeBridge": true}.

Local Docker note: if docker pull hangs, the macOS keychain credential helper is waiting (a pending keychain prompt). Pull with a config that has "credsStore": "".