I run a small fleet of self-hosted AI agents — they live in Slack, one per job, all of them on a homelab Kubernetes cluster spread across a few Proxmox servers. Their local brain is a pair of Tesla T40s: 24 GB datacenter GPUs, one in each of two physical servers, each passed through to a worker VM. Until this weekend the cards were a matched pair — each ran its own Ollama instance serving the same model, load-balanced, so losing one card meant losing capacity, not the fleet.
The problem was context. The model is a 27-billion-parameter Qwen quant whose weights want about 17.7 GiB, and a T40 with ECC on has about 22.5 GiB usable. Subtract the weights, subtract headroom for the Plex transcoder that shares the card, and the biggest context window that fits is 73,728 tokens. My agent framework has a hard floor of 64k. That’s barely above the floor, which meant the agents’ history compaction — the background process that summarizes old conversation to make room — ran constantly, each run an expensive, disruptive pause in the middle of real work.
Here’s what finally got under my skin: I had two of these cards. Forty-five gigabytes of combined VRAM, holding two identical copies of the same 17.7 GiB of weights. Load balancing does not pool memory. Two cards at 73k is two separate conversations of 73k. It is never one conversation of 147k.
One copy of the weights
llama.cpp has an answer for this that Ollama does not: an RPC backend. One llama-server process runs on node A and holds its local T40 directly; a small RPC worker on node B offers up the second T40 over the network. The server sees two devices and splits a single copy of the model across them — one set of weights, one KV cache. The second card stops being a redundant copy and becomes pure context budget.
The model’s architecture helps here. Only 16 of its 64 layers are full attention — the KV cache costs about 38 KiB per token — and the other 48 are linear-attention layers with a fixed 748 MiB state that doesn’t grow with context at all. Long context is unusually cheap on this design. Split across both cards, the model loaded at its full native window: 262,144 tokens, 3.6× the single-card ceiling, with about 12 GiB of VRAM still to spare. Two request slots sharing one unified pool.
Getting there cost most of a night I’d like back. The first attempt produced pure garbage at full GPU speed — every completion a run of one repeated character. I suspected the RPC link, the split, the cards. What ruled the hardware out was reproducing it with CPU-only inference: same garbage, no GPUs involved. A popular community quantization of the model was simply broken. The reference build of the exact same quant worked perfectly.
Once that was swapped, it worked. Correct output, stable, a quarter-million-token window on hardware I own. I moved the fleet’s routes over and went to bed feeling clever.
The wire is in the loop
The next day produced numbers. Prompt ingestion — prefill, the reading phase — ran at 550–600 tokens per second with one stream. Token generation ran at 15–20 tokens per second alone, and collapsed to about 7.4 per stream the moment two streams shared the machine. Prefill wasn’t immune either: it roughly halved whenever anything else contended for the cards — measured down to 390, at worst 220. A cold 67,000-token coding-agent prompt took about 110 seconds before the first token appeared.
The root cause isn’t a bug, and that’s the interesting part. It’s the shape of the workload meeting the shape of the network.
A transformer generates text one token at a time, and each token depends on the one before it. When the model is split across two machines, every single generated token requires the intermediate activations to cross the network at the split boundary, get processed on the far card, and come back before the next token can even begin. llama.cpp’s RPC backend moves those tensors over plain TCP on my 10-gigabit Ethernet LAN. No RDMA, no GPUDirect — every hop goes through both machines’ kernels and network stacks. So generation speed is bounded by network round-trip latency, not by either GPU. Watching the cards during a generation was the tell: both mostly idle, waiting on the wire.
Prefill mostly escapes this because the prompt is known in advance. All 67,000 tokens can be processed in large parallel batches — a few big transfers instead of thousands of tiny ones — which makes it a bandwidth problem, and 10GbE has plenty of bandwidth. Decode is the opposite shape: a sequential conversation held one word at a time. Prefill is mailing someone a book. Decode is dictating the reply over the phone and waiting for “got it” after every single word. Same network, same two machines, completely different physics — which is exactly why the interconnects real datacenters use for this, NVLink and RDMA fabrics, exist at all. RDMA’s job is to shave that round trip down to microseconds by letting the NIC write into memory directly, no kernel in the path. I didn’t have that. I had TCP.
Slow alone might have been survivable. What made it an incident was that my clients had opinions about slowness. The agent framework has a 180-second stream-staleness watchdog; a coding CLI I run auto-cancels and retries on its own timeout. Slow turns tripped timeouts, retries doubled the load on the two slots, and the extra load slowed everything further — a clean feedback spiral. Cancelled requests made it worse rather than better, because the cancellation didn’t propagate through the proxy in front of the model, so a dead request kept its slot busy for a while after its client had given up. And I’d stacked a hidden cost on top without knowing it: this model’s chat template silently ignores the documented flag for disabling chain-of-thought reasoning; the working control turned out to be an obscure template-level setting. Until I found it, the model was burning thousands of invisible reasoning tokens per turn — and the watchdog counts only visible output, so a long think-block looked exactly like a dead stream and got killed at precisely 180 seconds.
The felt experience in Slack: agents that used to answer in a few seconds took anywhere from 30 to 260. After about a day of live measurement, I rolled it back.
The measurement that settles it
The rollback was, by design, boring: one variable — the Ollama replica count — plus restoring two proxy routes. The llama.cpp stack stays dormant on the cluster, one number away from a retry. And it handed me the cleanest comparison of the whole exercise, because minutes after the switch I could run the same model on one card with no network in the loop: 32 tokens per second of generation, against the pool’s 15–20. Prefill came out identical — about 600 tokens per second both ways. The split itself had been costing half the generation speed with nobody else on the machine, and none of the reading speed. That asymmetry is the entire story in two numbers: the phase that batches over the wire didn’t care, and the phase that round-trips per token paid full price.
What this taught
The lesson is not “don’t split models across machines.” The split worked. The model was correct, the 262k window was real, and the memory arithmetic paid off exactly as promised — the second card genuinely became context instead of redundancy. What failed was that I put a commodity Ethernet round trip inside the innermost loop of token generation and expected the GPUs to still be the thing that mattered. Once you split a model, the interconnect is the computer. The GPUs are just the parts of it that were easy to buy.
It also taught me to measure the two phases separately. “Inference speed” is two workloads wearing one name: a parallel, bandwidth-hungry read and a sequential, latency-bound write. My setup was fine at one and hopeless at the other, and any benchmark that averaged them would have hidden exactly the thing that mattered.
The retry is already sketched: symmetric GPUs across the three AMD servers — one of the three currently has no card of this class — a 100-gigabit round-robin fabric between them, and ideally an RDMA-capable transport in the inference stack, so the per-token round trip stops going through two kernels. Then re-pool, and re-measure. The quarter-million-token window sat there working for a day, on hardware I own, before the wire took it back. I intend to have it again — once the network stops being the slowest part of the computer.