← All articles

Event streams and Kafka, with a metrics pipeline as the example

Updated Nov 10, 2026 · 4 min read

An event stream is a log of things that happened, written in order and kept for a while. Producers append to the end. Consumers read from wherever they left off. Kafka is the best-known example. Putting a log between the systems that make data and the systems that use it means neither has to wait for the other, and a slow or broken consumer never makes a producer drop anything.

A log, not a queue

A classic queue hands each message to one worker and deletes it once it’s done. A log keeps every message for a set time, often a few days, whether anyone has read it or not. Each reader just remembers a position in the log, called its offset. That one difference gives you three things. A consumer that falls behind catches up later. A new consumer can start from the beginning and replay history. And several different systems can read the same stream without getting in each other’s way.

appendreadAgentsIngestLogkeeps 3 daysWritersown offset
Ingest only has to append to the log. Writers read at their own pace, so a paused database means a backlog, not lost data.

Topics and partitions

A topic is one named stream, like metrics. It’s split into partitions, and each partition is its own ordered log on a different broker. A producer picks the partition by hashing a key, so every message with the same key lands in the same partition, in order. Order is only promised inside a partition. Two messages with different keys may be read in either order.

Partitions are how a topic scales. Each one is a sequential file that a broker can append to very quickly, and adding partitions spreads the writes over more disks and machines.

Consumer groups

Consumers that share a group name split a topic’s partitions between them. Each partition is read by exactly one consumer in the group, which keeps its order intact. If a consumer dies, the group hands its partitions to the others. A different group, say an alerting service, gets its own copy of every message and keeps its own offsets.

Delivery guarantees

A consumer reads a batch, does its work, then commits its offset so a restart knows where to begin. When it commits decides what a crash costs you.

GuaranteeHowA crash means
At most onceCommit the offset, then do the workThe batch in flight is lost
At least onceDo the work, then commit the offsetThe batch in flight is done twice
Exactly onceCommit the work and the offset together, in one transactionNothing, but it costs more and only works inside one system

Most pipelines choose at least once and make the work idempotent, so doing it twice has the same effect as doing it once. A metric point keyed by its series and timestamp is a good example. Writing it again overwrites the same value, so a replayed batch changes nothing.

A metrics pipeline on top

Metrics are the classic use. An agent on every host sends a point for each series every 10 seconds, all day, so the load is a steady firehose of small writes. Estimate it as hosts × series per host ÷ 10 seconds × bytes per point. Five thousand hosts with 500 series each is 250,000 points a second, or 25 MB a second at 100 bytes a point.

Rollups

Nobody charts a year at 10-second resolution. A rollup job reads raw points and writes one row per series per minute, then per hour, and old raw points are deleted after a few days. Each rollup row should hold values that combine exactly, which are the sum, the count, the min and the max. Averages don’t combine, because a quiet minute and a busy minute count the same. Percentiles don’t combine either, so to roll up a p99, store a histogram of values and add the histograms together.

ResolutionKept forPoints kept
10 seconds7 days60,480
1 minute30 days43,200
1 hour2 years17,520
A typical retention plan for one series.

Label cardinality

A series is one metric plus one set of label values. The number of series is the product of how many values each label can take. A latency metric with 50 endpoints, 5 status codes and 4 regions is 1,000 series. Add a user_id label with 100,000 users and it can become 100 million. Each active series costs memory in the database’s index, often a few KB, so one careless label can take down the whole metrics system.

endpoint × status × region1,000
… × user_id100,000,000
Series for one latency metric, before and after adding a user_id label. This uses a log scale, so equal steps in length mean multiplying, not adding, so bars that look close can be 1,000× apart.

Keep labels to small, fixed sets of values. Per-user or per-request detail belongs in logs and traces, which are built to store many unique values.