Logs are the thing everyone agrees they should centralise and nobody wants to pay for. The
usual outcome is that the noisy source drowns the useful one, retention gets cut to a week, and the
one event you needed is three days past the edge of the index.
I centralised three very different feeds into one log lake and gave each one its own budget,
which is the decision that makes the rest of it work.
Three feeds, three index sets
- Kubernetes — container logs from three clusters, shipped as structured
records from a log agent on every node. - Hypervisors — syslog from seven virtualisation hosts over TCP, with a
disk-assisted queue at the sender. These are machines we manage, so the sender can be trusted to
buffer. - Firewalls and network gear — syslog from three network consoles and every
device they have adopted. This one is UDP, cannot be retransmitted, and is the reason there is a
buffering relay in front of the whole system at all.
The load-bearing decision is that these never share an index. The Kubernetes
feed alone produces several gigabytes a day; the network consoles produce a handful of audit events.
A single shared budget would silently roll every firewall message out of the index within hours,
and the loss would be invisible — you would search for last Tuesday’s config change and simply find
nothing, with no error anywhere to tell you why.
Separate index sets, each capped by size rather than by time, mean the loud feed can only ever
evict itself.
The alert that fired forty-one times in fifteen minutes
Before the streams were separated, a keyword alert written for firewall events matched the word
“Blocked” anywhere in any feed. A container log line containing that word fired the security alert
forty-one times in a quarter of an hour.
The rule that came out of it is absolute: every alert is scoped to exactly one
stream. An unscoped keyword query matches all three feeds, and the noisiest feed decides
how often your security alert fires.
The same class of mistake caught me a second time, more embarrassingly: an alert for fatal
errors in container logs fired on the provisioning system’s own log line announcing the
creation of that alert. The alert’s first act was to alert on itself.
Timestamps five hours in the future
This is the failure I would most want someone else to avoid, because every dashboard said the
system was healthy while it was quietly useless.
The relay in front of the log lake was re-templating incoming messages into a standard syslog
format. The network consoles do not send standard syslog — they send an event format with its own
header. The relay could not parse it, so it fell back to using the receive time and its own
source address. Worse, a timezone mismatch put every one of those events about five hours in the
future.
A search in the log UI defaults to a window ending at “now.” Events stamped in the future are
outside that window. So the pipeline reported healthy, messages were arriving and being indexed,
and the dashboard was empty. “Logs are streaming” and “I can see nothing” were both true at the
same time.
The fix was to stop re-templating and pass the raw bytes through. The lesson is broader:
a relay that reformats is a relay that can silently lie about when something
happened.
Making a dashboard tell the truth
The first firewall dashboard searched every stream rather than its own, which meant its numbers
included container logs. It looked plausible. It was wrong by a factor of thousands, and nobody
would have noticed, because a dashboard showing a big number looks like a dashboard that is
working.
Alongside it, tens of thousands of stray messages had accumulated in the default stream —
records that matched no routing rule and therefore landed in the catch-all, where nothing searched
them.
Both of those are the same underlying problem: a query that runs and returns something is
not the same as a query that answers your question. Every panel now names its stream
explicitly, and there is a check for messages arriving in the default stream, because a growing
catch-all means a routing rule is missing.
What I would tell someone starting this
Verify at the receiver, never at the sender. Several times, a configuration change on a sending
device persisted correctly, read back correctly, and shipped nothing at all — in one case because
the console had two separate settings surfaces for the same feature and only one of them was live.
The only proof that logging works is a message you can find on the other end.