Skip to content
Jacob NolletteCloud & software engineer

Three of the servers at home each carry a 24 GB GPU. A model that needs 29 GB fits on none of
them. The servers are also joined by 100-gigabit cables that, until this week, carried nothing but
storage replication. I wanted to know whether those cables could turn two cards into one. They
can: the same model, split across two GPUs in two machines, ran at 29 tokens a second over the
cables and 14 over the ordinary network.

Why the last attempt failed

I had tried pooling GPUs across machines before, with a different serving stack, and it was
slow. The notes from that attempt blamed the network: every token crossed between machines
several times over plain TCP on a 10-gigabit switch, and the round trips added up. The notes ended
with a precondition for trying again: matching GPUs and a faster network.

Both had quietly arrived. A hardware swap left all three cards on the same chip. And the three
hosts already had dual-port 100-gigabit network cards, wired to each other directly, one cable per
pair, no switch. That ring existed to move storage replication off the main network. Nothing else
used it.

The speed of the cable was never going to be the point. Two machines exchanging small messages
thousands of times a second are limited by how long each exchange takes, not by how much the cable
can carry. The thing that shortens an exchange is RDMA, where the network card moves memory from
one machine to the other without the operating system handling each packet.

Getting a network card into a virtual machine

The Kubernetes workers are virtual machines, and the GPUs are passed through to them. A network
card shared through the hypervisor’s software bridge cannot do RDMA, so the card had to be passed
through as well. The hosts still need those ports for storage, so the whole card could not go.

The answer is SR-IOV: the card presents extra virtual copies of itself, each one a separate
device that can be handed to a virtual machine while the host keeps the original. I had an AI
agent do the work, from the infrastructure repository and a shell on the hosts. It created one
virtual function on each end of the cable between two of the hosts, while pinging across that
cable five times a second. No packets were lost and the storage cluster did not notice.

Passing the device in means a full stop and start of the worker, since the GPU is already
pinned to it. So each worker was drained, shut down, given the new device, and started, one at a
time, with a check of every monitored site before and after.

What went wrong

The first worker would not start. To keep the host from claiming the new
virtual device for itself, the agent had switched off automatic driver binding for it. That
setting also stops the passthrough driver from binding, so the hypervisor refused to start the
machine, four times. One more setting, naming the passthrough driver explicitly, fixed it. The
worker was down for about seven minutes instead of three.

The first ping across the new link failed, and the link was fine. The agent
had given the two ends addresses from a four-address block and picked the last one for one side.
The last address of a block is the broadcast address. Several minutes of packet captures on both
hosts, all showing a working cable, ended with a one-character fix.

An unrelated portal went down for fifteen minutes. Draining the second worker
moved our remote-desktop portal’s manager process, and it took a quarter of an hour to report
ready on its new machine while everything that depends on it waited. It came back without
help. None of the public sites changed.

The numbers

First the link by itself, from inside one worker: a message took 1.5 microseconds to cross,
and bulk transfers ran at about 85 gigabits a second. A ping over the same cable took 260
microseconds, so the operating system’s network stack was costing more than a hundred times what
the wire did.

Then GPU to GPU, using the library that machine-learning frameworks use to move data between
cards. Over the ordinary network it moved 4.9 gigabits a second. Over the 100-gigabit cable with
plain TCP it moved 5.6, which is the result that surprised me most: a cable ten times faster
bought almost nothing. With RDMA on that same cable it moved 44.

Then the real test. A 14-billion-parameter model at full precision is 29 GB, too large for
either card. Served across both, with each layer’s work divided between the two GPUs:

Ordinary network RDMA over the cable
One conversation 14 tokens/s 29 tokens/s
Eight at once 71 tokens/s 177 tokens/s

Twice as fast for one person, two and a half times for eight. Dividing the model the other
way, with the first half of the layers on one card and the second half on the other, was slower on
the same cable: 17 tokens a second.

The feature that made it slower

RDMA has a further step for GPUs, where the network card reads and writes GPU memory directly
and the data never touches the machine’s main memory. It is the configuration every guide
recommends. I expected it not to work inside a virtual machine at all. It worked, and it cut the
transfer rate from 44 gigabits a second to 27.

The likely reason is where the cards sit. On these hosts the GPU and the network card hang off
different processors’ memory regions, and inside a virtual machine every direct transfer between
them goes through a translation layer. Copying through main memory first turned out to be the
shorter road. I would not have guessed that, and I would have left the recommended setting on if
the agent had not measured both.

What is not done

This is a proof on two of three machines. The third needs the same treatment, and it hosts
the workstation most of this infrastructure is managed from, so it waits for a quiet moment.

The test containers ran with full privileges on the host’s network, which is fine for an
evening and wrong for anything permanent. A proper device plugin would hand the network device to
a container the way the GPU already is.

A pooled model wants both cards to itself, and those cards currently hold the model our agents
fall back to. Whether one larger model beats two copies of a smaller one is a question about how
the fleet is used, not about cables.

And the virtual devices on the hosts were put there by hand, with a small boot script so a
restart does not strand the workers. They are not yet in the automation that built everything
around them. That is the next change, before anything else is stacked on top.