Web crawlers, the URL frontier and crawling politely
Updated Nov 10, 2026 · 4 min read
A web crawler starts from a list of seed URLs, fetches each page, pulls out its links, and adds the new ones to the list. Repeat that a few billion times and you have a copy of the web. The loop is simple. The hard parts are doing it fast without overloading anyone’s site, not wasting work on pages you already have, and not getting stuck.
The frontier
The frontier is the set of URLs waiting to be fetched. It decides two things, what to fetch next and when. The classic design, from the Mercator crawler, splits those jobs in two.
- Front queues set priority. Important pages, such as home pages and pages many others link to, go into higher-priority queues and get picked first.
- Back queues set politeness. There is one queue per site, and each has a “not before” time. A fetcher takes the site that is due soonest, fetches one URL, and pushes that site’s time a little into the future.
With back queues, each site sees a gentle trickle while the crawler as a whole runs as fast as there are sites ready. The frontier is far too big for memory, so queues live on disk with only their heads in memory, and the whole thing is split across machines by hashing the site name. Each site then belongs to exactly one machine, which keeps its pacing in one place.
Being polite
- Read
robots.txtbefore crawling a site, cache it for a day or so, and skip any disallowed path. It became a standard, RFC 9309, in 2022. - Keep one request at a time per site, with a gap between them. Some sites ask for a gap with
Crawl-delay. It isn’t part of the standard, but it’s polite to honor it. - Back off when a site answers
429or503, or simply slows down. - Send a clear
User-Agentwith a URL that explains who you are.
Not fetching the same URL twice
Pages link to each other constantly, so most links you find point somewhere you have already been. First normalize each URL. Lowercase the host, drop the #fragment, remove tracking parameters such as utm_source, and sort the query string. Then check it against every URL you have seen.
Tens of billions of URLs at about 80 bytes each is terabytes, too much to keep in memory. A Bloom filter answers “have I seen this?” in a fraction of the space. It sets a few bits for each URL. A new URL might find all its bits already set by others, so “seen” is sometimes wrong. “Not seen” never is. For a crawler that means a few new pages are skipped, and none are fetched twice.
| False positives | Bits per URL | For 10 billion URLs |
|---|---|---|
| 10% | 4.8 | 6 GB |
| 1% | 9.6 | 12 GB |
| 0.1% | 14.4 | 18 GB |
| Exact list of URLs | ≈ 640 | ≈ 800 GB |
Not storing the same page twice
Different URLs often return the same page, from mirrors, syndicated articles, or printer-friendly copies. Hash each page’s content and keep a set of hashes. If you already have it, skip parsing and storing it. Near-duplicates, like the same article with a different date in the footer, need a fingerprint such as SimHash, where similar pages get hashes that differ in only a few bits.
Crawler traps
Some sites generate links without end. A calendar links to next month forever. Session ids in URLs make every visit look like a new page. Faceted search combines filters into millions of URLs. None of these is malicious, and all of them can eat your crawl.
- Cap pages per site and link depth from the site’s home page.
- Cap URL length, and treat URLs that repeat a path segment (
/a/b/a/b/a/b) with suspicion. - Watch for sites where almost every page is new by URL but a duplicate by content.
Sizing a crawl
| Question | Working | Answer |
|---|---|---|
| Pages a second for 1 billion a month | 1,000,000,000 ÷ 2,592,000 s | ≈ 390 pages/s |
| Sites in flight at one request a second each | Same as pages a second | ≈ 390 sites |
| Storage for a billion 100 KB pages | 10⁹ × 100 KB | 100 TB, about 25 TB compressed |
| Bandwidth at 390 pages/s of 100 KB | 390 × 100 KB × 8 | ≈ 310 Mbps |
DNS is the surprise bottleneck. Every new site needs a lookup, and public resolvers will not thank you for thousands a second. Big crawlers run their own caching resolvers.
Recrawling
The web changes under you. A news front page changes every few minutes and an old blog post never. Track how often each page changed when you revisited it, and schedule the next visit to match, with important pages checked more often. Conditional requests with If-Modified-Since or ETag let an unchanged page come back as a tiny 304 Not Modified.