I run a small fleet of AI agents that live in Slack — one per function, each with its own personality and toolset, all self-hosted on a homelab Kubernetes cluster. One of them, Atlas, got a new job this month: hold an actual spoken conversation in a Discord voice channel, in real time, entirely on hardware I own. No cloud speech API in the loop, no audio leaving the network.
That constraint was deliberate. The framework these agents run on offers three ways to do voice: hand the whole conversation to a cloud speech-to-speech model, proxy through an external agent, or run a local speech-to-text → chat → text-to-speech pipeline. The cloud option would have been faster to ship. I chose the local pipeline anyway, because the fleet already owned a transcription service for Slack voice notes, and running two different voice mechanisms — one cloud, one local — for one small feature wasn’t worth the operational surface. One mechanism, everywhere.
The pipeline, once assembled: you speak, a local Whisper model transcribes it, a local language model thinks, a local text-to-speech model speaks the reply. All four steps run on two repurposed GPUs sitting in the same rack as everything else. Getting that loop working was the easy part. Getting it to feel like a conversation instead of a phone call with a two-second delay on every line — that took two weeks, and every fix uncovered a new bottleneck hiding behind the one just fixed.
The 13-second reply that wasn’t the model’s fault
The first real test produced a six-word reply that took thirteen seconds to come back. For a text-to-speech model with 82 million parameters, that’s absurd — it should be near-instant. The obvious diagnosis was "wrong hardware, move it to a bigger card." That would have been wrong twice over: the bigger cards were deliberately reserved for something else, and the card actually in use wasn’t the problem at all.
The real cause was stranger. The speech pod had been running for twelve days. A background service it depended on — the thing that hands out GPU access inside the cluster — had restarted twenty-one hours earlier. The pod’s connection to the GPU quietly went stale when that happened, but nothing crashed. The library reporting on GPU status inside the pod failed silently, while the speech engine itself kept claiming it was running on the GPU and just… wasn’t. It had been doing all of its work on the CPU instead, unnoticed, for the better part of two weeks, because "slow but technically working" doesn’t look like a failure from the outside. Restarting the pod took generation time from thirteen seconds to under half a second — a 35x improvement with zero configuration changes, just re-establishing a connection that had silently died.
That became a standing rule for the whole fleet: a GPU workload that’s inexplicably slow is worth checking for a dead connection before you go looking for a slower cause.
Chasing latency with a stopwatch, not a hunch
With the pipeline actually running on GPU, the next round was about shaving real seconds off a real conversational turn. This is where the work turned into genuine measurement rather than guessing:
- A bigger context window fixed a compaction problem, unintuitively. The language model was periodically pausing mid-conversation to compress its own history — an expensive, disruptive operation — because its working memory was set far smaller than the model actually supported. Raising the ceiling didn’t cost meaningfully more GPU memory (the model’s attention mechanism made a larger window nearly free), and it cut how often that compression kicked in by roughly seven times.
- A smaller, faster model made things worse, not better. The obvious lever for "make replies faster" is "use a smaller model." Tested in isolation, the smaller model looked twice as fast. Tested for real, inside the actual agent with its full toolset, it started returning malformed responses that the system then had to retry — turning a hoped-for speedup into some of the slowest turns of the whole project. A synthetic benchmark with two tools available doesn’t predict behavior against three dozen.
- The real cause of a 7-second delay turned out to be two hidden identifiers. Every request to the language model carried a system prompt that ended with a pair of unique session IDs, regenerated on every call, sitting at the very end of an otherwise-identical 30,000-character block of text. Because they sat at the end, and because language models process prompts left to right, those two IDs invalidated the model’s cached understanding of everything before them — all thirty thousand characters, on every single turn. Removing the disposable IDs from that position let the model reuse its cache instead of re-processing the entire prompt from scratch, cutting seconds off nearly every reply. Proved by replaying the exact same request twice, changing only those 72 characters, and watching the processing time jump from 19 milliseconds to nearly 6 seconds.
- The reply itself was shorter than the audio needed to say it. Once the thinking and transcription steps were both fast, the single largest remaining chunk of every turn was simply how long it took to play the spoken reply out loud. Speeding up playback by 20% — imperceptible as "sped up," noticeably brisker in practice — shaved real time off every single response, cloud or hardware cost be damned, because it was free.
- A quieter model can be a slower one. The smaller model tried a second time turned out to generate a lengthy internal reasoning pass before every reply — dozens of extra tokens of "thinking out loud" that never got spoken, but had to be generated anyway. Turning that off cut the dead air before the agent started speaking.
Each of these individually shaved a few tenths of a second to a couple of seconds. Together, a conversational turn that started at somewhere around thirteen seconds landed closer to four.
The bug that made it look broken for a week
The most humbling fix had nothing to do with speed. After all of the above was tuned and verified, Atlas simply never answered in the voice channel — every spoken message was heard, transcribed correctly, and then silently discarded before it ever reached the language model.
The cause: an authorization check meant to restrict who the agent would listen to had been configured with the wrong ID. Discord and Slack identify the same person with two completely different identifier systems, and the voice channel’s allow-list had been populated with the Slack version by copy-paste. Every incoming voice segment failed that check and was dropped — correctly transcribed, silently discarded, logged only to a debug file that isn’t visible in normal container logs. It read, from the outside, exactly like the newly-swapped language model had regressed, because both changes landed in the same week. The model was fine the entire time; nothing had ever been listening.
A second, similarly quiet bug hit the Slack side of the same voice feature days later: a "no mention required, every message is addressed to you" behavior that had been true in the agent’s written instructions for months, but never actually configured in the system that enforces it. It had looked like it worked, because nearly every message that reached the agent had been auto-mentioned by the Slack client’s autocomplete before anyone noticed — until a voice clip arrived, which structurally can’t carry an @-mention, and hit the one code path that had never been exercised.
What two weeks of this actually taught
Nothing here was one big bug. It was the same small failure mode, over and over, in different clothes: something that looks like it’s working — a GPU handle that isn’t erroring, a config block that reads correctly, an allow-list that seems populated — while quietly not doing its job underneath. The fix, every time, was resisting the urge to trust that silence means success, and instead measuring the actual thing: time the real turn, replay the real payload, watch the real log line. The felt experience of talking to Atlas now — a few seconds of latency instead of ten-plus, replies that land promptly instead of trailing off into a long silence — is the sum of about a dozen small, boring, verified fixes, not one clever one.