← All articles

Job schedulers, leases and running a job exactly once (almost)

Updated Nov 10, 2026 · 4 min read

Almost every product has work that happens outside a request. Reminder emails, nightly reports, retried webhooks, a video to transcode. A job scheduler stores that work, decides when each job is due, and hands it to a fleet of workers. The hard part isn’t running jobs. It’s running each one on time and only once when workers crash, freeze and lose the network halfway through.

The shape of a scheduler

Jobs live in durable storage, usually a database table with a row per job. Each row has an id, what to run, when it’s due, how many times it has been tried, and who owns it right now. Workers claim due jobs, run them, and mark them done. Nothing about a job lives only in a worker’s memory, so any worker can die without losing work.

addleaseYour appadds jobsJobs tableby due timeWorkersclaim and runDead lettersfailed every try
Jobs wait on disk. Workers claim them with a lease and keep it alive with heartbeats.

Leases and heartbeats

When a worker claims a job it takes a lease, which is ownership with an end time. The row says “worker 7 owns this until 10:04:30”. While the job runs, the worker sends a heartbeat every few seconds that pushes the end time out. A long job keeps its lease for as long as it needs.

If the worker dies, the heartbeats stop and the lease runs out. A sweeper, or the next worker looking for work, sees the expired lease and makes the job available again. Nobody needs a message from the dead worker. The silence is the signal.

The lease length is a trade. At worst, a dead worker’s job waits one lease length plus however long the sweeper takes to look. Short leases free jobs sooner. But a worker that freezes for longer than the lease, in a long garbage collection pause or a slow network, loses a job it’s still running, and now two workers have it.

LeaseDead worker’s job waitsRisk of a live worker losing its job
10 sAbout 10 sHigh. One long pause is enough
60 sAbout a minuteLow
10 minUp to 10 minutesVery low, but recovery is slow

At least once, so make jobs idempotent

There’s no way to promise a job runs exactly once. A worker can finish the work and die before it marks the job done, and then the job has to run again. Queues and schedulers deliver at least once. The “exactly once” you see from outside comes from making the job idempotent, so a second run changes nothing.

  • Store the job id with the effect, in the same transaction, under a unique key. A second run finds the id and stops.
  • Write absolute values, not changes. “Set the plan to Pro” is safe to repeat. “Add 100 credits” isn’t, unless it’s keyed.
  • Pass an idempotency key to outside APIs, like a payment provider, so they spot the repeat too.
  • Use a fencing token, a number that rises with every new lease. Storage rejects writes carrying an older number, so a worker that wakes from a long pause can’t overwrite newer work.

Running cron once across many nodes

Recurring jobs like “every night at 2:00” are easy on one server and surprisingly hard on several. Run cron on every node and the job runs once per node. Run it on one node and that node is a single point of failure.

The usual answer is leader election. The nodes compete for a lease in a coordination store such as etcd, ZooKeeper or a database row. The winner is the leader, and only the leader turns schedules into jobs. It renews its lease like a worker sends heartbeats. If it dies, the lease lapses and another node takes over within seconds.

Jobs for later

A reminder due next March shouldn’t sit in a message queue. Most queues cap how long they’ll hold a message back. Amazon SQS, for example, allows at most 15 minutes. Keep far-off jobs in the table with an index on due time, and have workers poll for the earliest due rows in small batches. In Postgres, SELECT … FOR UPDATE SKIP LOCKED lets many workers poll the same table without grabbing the same rows. Some systems keep the same idea in a Redis sorted set scored by due time.

Retries, backoff and dead letters

Jobs fail because something they call is down. Retrying straight away hammers a service that’s already struggling. Exponential backoff waits longer after each failure, say 1, 2, 4 and 8 seconds. Jitter adds a random part to each wait, so a thousand jobs that failed in the same second don’t all retry in the same second.

Some jobs will never succeed, because of bad input or a deleted account. After a set number of tries, move the job to a dead-letter queue instead of retrying forever, and alert someone. A growing dead-letter queue is one of the best early warnings you have.

Sizing the fleet

Little’s law says the number of jobs running at once equals the rate they start times how long each takes. 1,000 jobs a second that take 4 seconds each means 4,000 running at once. If a worker runs 20 at a time, that’s 200 workers at that rate, before any headroom.

running at once = jobs per second × seconds per job
workers = running at once ÷ jobs per worker