I run a Jitsi Meet server at home. For two months it was reachable only from the house network
and a few of my own devices, which is fine until you want to talk to someone else. A friend,
another software engineer, and I wanted to use it. Getting him onto it took one evening. Making
the call good took a second problem I did not know I had: a 70 ms round trip to the video bridge,
with every packet of mine going through a relay in Chicago.
Jitsi over Tailscale, in three attempts
Attempt one was the public internet. Tailscale Funnel publishes a machine on
your tailnet to the world, with a certificate, and no ports opened at home. It has two limits that
matter for video. It serves only the machine’s own ts.net name, and it carries TCP
only, on three ports. WebRTC media is UDP. So the page and the signalling went through Funnel
directly, and the media needed a TURN relay: coturn, sitting behind a second Funnel port, taking
each browser’s media in over TLS and handing it to the video bridge as UDP.
A public TURN server is a relay into whatever network it sits on, so that one got fenced
before it got deployed. It listened on loopback only, refused every peer address except the
bridge, and accepted only the short-lived credentials the meeting server hands out. I tested the
exact flags first: traffic to the bridge went through, traffic to a public address and to a
neighbouring private one was refused, and a wrong secret could not allocate anything. The test
also caught a flag the current coturn release no longer accepts, which would have been a crash on
deploy.
It crashed on deploy anyway, for a different reason. The coturn binary carries a file
capability, and the container was set to drop all capabilities and forbid privilege escalation.
The kernel refuses to run a file-capped binary under those two settings together. Docker’s
defaults had hidden that in my local test. One capability added back, and it started.
Attempt two was not public at all. Funnel worked, verified from outside. But
a meeting server for a handful of known people does not need to face the internet. Tailscale has
a feature for the smaller job: share one machine into someone else’s tailnet. He accepts an
invite and gets that machine and nothing else of mine. Funnel went off, which took about ten
seconds and no restart, and the same two ports stayed open to the tailnet only.
Attempt three was removing my own gate. The first real call sat at “waiting
for moderator”. Rooms required a host to sign in through an identity provider, the sign-in
callback had moved to the new address with the server, and I had not registered the new address
with the provider. Mid-meeting was the wrong time to fix that, so host sign-in came off. With the
server reachable only from the tailnet and from machines I have shared it with, the network is
the gate.
The call worked, and it was choppy
The bridge reported a 70 ms round trip and about 250 kbps each way. The obvious suspect was
Wi-Fi, since we were both on it. It was not Wi-Fi.
Tailscale tries to connect two devices directly, punching through each side’s router. When it
cannot, it falls back to DERP, its relay network. DERP always works, which is the point of it, but
it is TCP through a datacenter. My machine’s connection to the house was on a DERP relay. For
media that is the worst case: one lost packet stalls everything queued behind it, and my media was
already inside a TCP relay of my own making.
The wrong turn was ten days old, and it was mine
The house’s Tailscale subnet router used to be a small virtual machine. Ten days before this
call I had moved it into the Kubernetes cluster, managed by Tailscale’s operator. One fewer
machine to patch, everything declared in one place. The Jitsi server’s own tailnet presence was a
container in the cluster too.
What I had not checked was what the internet looks like from inside a pod. A pod’s traffic
leaves through the cluster’s own outbound NAT first, and then through the house router. Two
layers, and the inner one picks a different source port for every destination.
tailscale netcheck answered it in one line. From a normal pod,
MappingVariesByDestIP was true. From a pod on the host’s network,
skipping the cluster’s NAT, it was false. Hole punching depends on a device
showing the same public port to everyone. With a port that changes per destination, a peer behind
an ordinary home router has nothing stable to aim at, so Tailscale gives up and relays.
The house router was innocent. Kubernetes had added the hard NAT, and I had put the two
components that most need a clean path to the internet behind it.
Putting the router back outside
The fix was to undo the tidy-up: a virtual machine again, outside the cluster, on the network
segment that already existed for edge boxes. The zone firewall still had the rule allowing
Tailscale’s port to that address, left over from the old machine.
I had an AI agent do the build, working from the infrastructure repository. It wrote the
playbook that creates the machine from a pinned cloud image and the one that configures the guest,
then brought the new router up beside the old one. Before cutting over it tested fifteen addresses
across the house network from both routers and got identical results. Then it removed the
in-cluster router, and the new machine took over all seventeen routes in about five seconds.
From the new machine, netcheck read false, and the house router offered it automatic port
mapping on top. Every active connection to it was direct, including mine. On the call still in
progress, the bridge’s round trip went from 70 ms to 18.5 ms, with no relayed participants.
One thing turned up on the way that had nothing to do with video. The new machine was in no
backup job. The backup class for the router was still declared, but its pool had disappeared with
the old machine, and the catch-all job skipped that machine’s ID because the class was supposed to
cover it. It would have been backed up by nothing, with every job reporting success.
What is not fixed
The Jitsi server’s own tailnet presence is still a container behind the same two layers. My
devices no longer care, because they reach the bridge through the new router. A guest who joins
through a share does not get those routes, so a guest’s media still goes through my TURN relay and
can still land on DERP. Moving that entry point out of the cluster is the next piece. Until then I
have fixed this more thoroughly for myself than for the person I was talking to.
The lesson I am keeping is narrow. Before moving anything that other networks must reach
directly into a cluster, run netcheck from where it will actually live. It takes ten seconds, and
it would have saved the ten days.