I run a five-node Proxmox cluster with hyperconverged Ceph and a three-node Kubernetes control plane on top of it. It has been up for over a year. Nothing crashes, deployments roll, pods schedule.
It was just slow, in a way I could never quite point at. A kubectl apply took a beat longer than it should. Controllers reconciled with a lag you only noticed if you were watching for it. The kind of slow that gets absorbed into “that’s just how it is” and never becomes a ticket.
The number underneath that feeling turned out to be ugly. Mean etcd write-ahead-log fsync of 24 to 64 milliseconds, sustained, all day, every day. etcd’s own documentation puts the healthy ceiling at 10 ms. I had been running at five times that for the life of the cluster.
This is the story of what was actually causing it, and of the two hours I spent confidently fixing the wrong thing first.
The theory that felt obvious
Ceph was mid-drain on one node, backfilling slowly because the recovery throttles were set deliberately low. That looked like the answer: slow recovery means a long window of sustained interference on the same drives etcd was using. Raise the throttles, finish the drain sooner, get the interference over with.
So I raised them. Concurrent backfills went from 4 to 29. Recovery throughput went from about 20 MiB/s to 409 MiB/s.
That looked like a win for roughly ninety seconds.
Then etcd fsync p99 hit 1.5 seconds, against a 5-second election timeout. The API server started failing its own startup probes. Controller-manager leader election flapped three times in two minutes. And the layer-2 speakers that announce every external LoadBalancer address restarted — they only run on the control-plane nodes, so every external service IP in the cluster went dark at once.
In-cluster traffic never noticed. A pod curling the same services got HTTP 200 the entire time, which is its own lesson: reachability from inside the cluster is not evidence that anything is fine.
The number that killed the theory
I backed the throttles off by half, expecting the latency to recover roughly in proportion.
It did not. Recovery dropped from 409 to 216 MiB/s — cut exactly in half, as asked. Fsync p99 moved from 1522 ms to 1020 ms.
That ratio is the whole story, and I nearly missed it because a 30% improvement still looks like progress. If the problem were contention for bandwidth, halving the load would roughly halve the latency. It didn’t, because the constraint was never volume.
These are consumer NVMe drives with no power-loss protection. Without a protected write cache, every O_DSYNC commit has to flush all the way to NAND before it can be acknowledged. That cost is per operation. You cannot throttle your way underneath a per-operation floor. You can only stop asking for the operation.
Nine copies of every write
Once I stopped looking at the throttles, the architecture was embarrassing.
etcd is a replicated database. That is the entire point of it. Three members, Raft consensus, every write acknowledged by a majority before it commits. The data is already on three machines by design.
Each of those three members was storing its data on a Ceph pool with size 3. So each of the three Raft copies was itself being written to three OSDs.
Three times three. Nine copies of every key, and a network round trip to the slowest of three drives before Raft could even begin its own round of agreement.
The second replication layer bought nothing. Ceph’s durability guarantee is redundant with Raft’s — if a member’s disk dies, etcd’s answer is to remove that member and add a fresh one, which it does routinely and fast. Ceph was not protecting me against a single failure that etcd could not already survive.
What it did buy was latency, on the one write path in the whole cluster where latency is load-bearing. Every lease renewal, every leader heartbeat, every object write in the Kubernetes API goes through that fsync.
The fix was to take something away
Each hypervisor already had about 700 GB free in its boot volume group. I made a local LVM storage pool, put one control-plane VM on each host, and moved each virtual disk off Ceph onto the local disk underneath it.
No new hardware. No OSD removed. No data moved between machines. The fix was subtraction.
The cost is real and worth stating plainly: those three VMs can no longer live-migrate. Their disks are node-local now, so host maintenance means shutting a member down instead of moving it. For a three-member Raft cluster that tolerates losing one, that is a fine trade. For a two-member cluster it would not be — which is its own argument against ever running two.
One trap on the way out
If you are on Proxmox, this one will cost you an hour.
Moving a disk onto plain LVM fails outright while the drive is using aio=io_uring, because that pairing is known-bad and Proxmox guards against it. Good.
But the guard only fires on the move. A disk already sitting on LVM in that state raises nothing at all — no warning, no log line, no hint in the UI.
Two of my three control-plane nodes had been moved earlier without an explicit aio setting, which left them on the io_uring default. They had been quietly running the exact combination the guard exists to prevent, and it showed up as measurably worse fsync than the third.
The check validates the transition, not the state. That is worth remembering anywhere you rely on a guard rail: it may only be watching the door, not the room.
What it measured
Three independent measurements, all taken while Ceph was actively backfilling 2.1 TiB in the background — so if anything these are conservative.
| Measurement | On Ceph | On local disk |
|---|---|---|
| API server mutating requests, mean | 43–65 ms | 9.6 ms |
| etcd WAL fsync, mean | 24–64 ms | 1.7–3.1 ms |
| etcd WAL fsync, p99 | 124–1522 ms | 4–16 ms |
| Guest disk write latency | 13–16 ms | 0.5 ms |
| etcd endpoint round trip | 34 / 48 / 636 ms | 11 / 12 / 15 ms |
Between five and thirty times faster depending on where you put the probe. The API server number is the one a human feels, because everything in Kubernetes goes through it.
The p99 row is the one that changed character rather than just magnitude. A multi-second tail on the control plane is not a slow cluster. It is a cluster that is one bad minute away from a leader election. That tail is simply gone now — it has not moved off 9.6 ms since the last disk finished copying.
What did not get faster
Ceph is unchanged. The physical hosts show no step change in disk latency at all. Every workload still on the shared pool is riding exactly the same drives at exactly the same speed it was yesterday.
The control plane got faster. Because everything in Kubernetes is mediated by the control plane, that reads to a person as the whole cluster getting faster — but the honest claim is narrower than the feeling, and I want to be precise about which one I am making.
And 9.6 ms is not the floor. These are still DRAM-less consumer drives with no power-loss protection. Going local removed the network hop and the extra replication, which was most of the 30 ms, but every commit still flushes to NAND. Single-digit milliseconds is as low as this hardware goes. One small enterprise SSD with a protected write cache per host is the only change that removes the ceiling instead of lowering it.
One member is still consistently twice as slow as the other two, and that one tracks host memory rather than disk — its hypervisor is 20 GB into swap. Another day.
What I would tell myself two hours earlier
Shared storage is the default in a hyperconverged cluster, and defaults are exactly how this happens. Every VM goes on Ceph because every VM goes on Ceph. Nobody sits down and decides that the consensus database should have a second consensus system underneath it. It is just where virtual disks live.
The question worth asking of any storage layer is what failure it protects against that nothing above it already handles. For etcd on a replicated pool, the answer is genuinely nothing, and you pay for it on every single write.
The other one is about the shape of evidence. Halving the load and getting a 30% improvement was the moment my contention theory should have died. I kept it alive for another twenty minutes because a 30% improvement still feels like I was onto something.
When the response to cutting load in half is that far from proportional, you are not contending for a resource. You are paying a fixed cost per operation. And the only fix for a fixed cost per operation is to stop issuing the operation.