Event streams and Kafka, with a metrics pipeline as the example
Updated Nov 10, 2026 · 4 min read
An event stream is a log of things that happened, written in order and kept for a while. Producers append to the end. Consumers read from wherever they left off. Kafka is the best-known example. Putting a log between the systems that make data and the systems that use it means neither has to wait for the other, and a slow or broken consumer never makes a producer drop anything.
A log, not a queue
A classic queue hands each message to one worker and deletes it once it’s done. A log keeps every message for a set time, often a few days, whether anyone has read it or not. Each reader just remembers a position in the log, called its offset. That one difference gives you three things. A consumer that falls behind catches up later. A new consumer can start from the beginning and replay history. And several different systems can read the same stream without getting in each other’s way.
Topics and partitions
A topic is one named stream, like metrics. It’s split into partitions, and each partition is its own ordered log on a different broker. A producer picks the partition by hashing a key, so every message with the same key lands in the same partition, in order. Order is only promised inside a partition. Two messages with different keys may be read in either order.
Partitions are how a topic scales. Each one is a sequential file that a broker can append to very quickly, and adding partitions spreads the writes over more disks and machines.
Consumer groups
Consumers that share a group name split a topic’s partitions between them. Each partition is read by exactly one consumer in the group, which keeps its order intact. If a consumer dies, the group hands its partitions to the others. A different group, say an alerting service, gets its own copy of every message and keeps its own offsets.
Delivery guarantees
A consumer reads a batch, does its work, then commits its offset so a restart knows where to begin. When it commits decides what a crash costs you.
| Guarantee | How | A crash means |
|---|---|---|
| At most once | Commit the offset, then do the work | The batch in flight is lost |
| At least once | Do the work, then commit the offset | The batch in flight is done twice |
| Exactly once | Commit the work and the offset together, in one transaction | Nothing, but it costs more and only works inside one system |
Most pipelines choose at least once and make the work idempotent, so doing it twice has the same effect as doing it once. A metric point keyed by its series and timestamp is a good example. Writing it again overwrites the same value, so a replayed batch changes nothing.
A metrics pipeline on top
Metrics are the classic use. An agent on every host sends a point for each series every 10 seconds, all day, so the load is a steady firehose of small writes. Estimate it as hosts × series per host ÷ 10 seconds × bytes per point. Five thousand hosts with 500 series each is 250,000 points a second, or 25 MB a second at 100 bytes a point.
Rollups
Nobody charts a year at 10-second resolution. A rollup job reads raw points and writes one row per series per minute, then per hour, and old raw points are deleted after a few days. Each rollup row should hold values that combine exactly, which are the sum, the count, the min and the max. Averages don’t combine, because a quiet minute and a busy minute count the same. Percentiles don’t combine either, so to roll up a p99, store a histogram of values and add the histograms together.
| Resolution | Kept for | Points kept |
|---|---|---|
| 10 seconds | 7 days | 60,480 |
| 1 minute | 30 days | 43,200 |
| 1 hour | 2 years | 17,520 |
Label cardinality
A series is one metric plus one set of label values. The number of series is the product of how many values each label can take. A latency metric with 50 endpoints, 5 status codes and 4 regions is 1,000 series. Add a user_id label with 100,000 users and it can become 100 million. Each active series costs memory in the database’s index, often a few KB, so one careless label can take down the whole metrics system.
Keep labels to small, fixed sets of values. Per-user or per-request detail belongs in logs and traces, which are built to store many unique values.