URL Frontier

TLDR: The URL frontier is the data structure holding every URL a crawler has discovered but not yet fetched. It decides what gets crawled next and enforces politeness between requests to the same host.

The URL frontier is the queue at the center of any web crawler. Discovered URLs enter it, the crawler pulls URLs out of it to fetch, and newly found links are fed back in. In a trivial crawler the frontier is a simple list. In a production crawler it is a sophisticated component that must handle prioritization, politeness, deduplication, and persistence across billions of URLs.

What the Frontier Must Handle:

1. Ordering: Deciding which URL to fetch next, whether by discovery order, priority score, or freshness requirement.
2. Politeness enforcement: Ensuring consecutive requests to the same host are spaced apart, even when thousands of that host’s URLs are queued.
3. Deduplication: Rejecting URLs already crawled or already queued, after URL normalization collapses equivalent variants.
4. Persistence: Surviving crawler restarts without losing queued work, which matters when a crawl runs for days.
5. Scale: Holding far more URLs than fit in memory, which requires spilling to disk or a distributed store.

The Two-Queue Design:

The classic frontier architecture, described in Stanford’s Introduction to Information Retrieval, separates the problem into two queue layers.

Front queues handle prioritization. Each front queue holds URLs of a given priority level, and a selector pulls from higher-priority queues more often. Priority might reflect page importance, update frequency, or business value.

Back queues handle politeness. Each back queue holds URLs for exactly one host. A heap tracks the earliest time each host may next be contacted. The crawler always fetches from whichever host is due next, guaranteeing that crawl delay is respected without stalling the whole crawler while one host waits.

This separation is what lets a crawler be simultaneously fast overall and polite to each individual server.

Frontier States:

DISCOVERED  → URL extracted from a page, not yet validated
QUEUED      → passed scope and dedup checks, waiting in frontier
IN_FLIGHT   → currently being fetched
COMPLETED   → fetched successfully, links extracted
FAILED      → fetch failed, may be retried with backoff
SKIPPED     → rejected by scope rules, robots.txt, or depth limit

Tracking state explicitly prevents the two most common frontier bugs: refetching URLs already processed, and losing URLs that failed mid-flight when a worker crashed.

Deduplication at Scale:

The frontier must answer “have I seen this URL before?” for every discovered link, often billions of times per crawl. Storing full URLs in a hash set becomes impractical past a certain size.

Production crawlers typically hash each normalized URL to a fixed-length fingerprint and store only fingerprints. Bloom filters are common where a small false-positive rate is acceptable, since they use dramatically less memory than exact sets. The tradeoff is that a false positive silently drops a URL that was never actually crawled, so systems requiring complete coverage pair a Bloom filter with an exact backing store.

Distributed Frontiers:

When a crawl spans multiple machines, the frontier must be shared or partitioned. Partitioning by host is the standard approach: a hash of the hostname determines which worker owns that host’s URLs. This keeps all politeness logic for a given host on a single machine, which is what makes per-host rate limiting enforceable.

The alternative, a centrally shared frontier, is simpler to reason about but becomes a throughput bottleneck and a single point of failure. Most large-scale systems partition and accept the added complexity of routing discovered URLs to the correct owner.

Frontier and Crawl Outcomes:

Frontier design determines what a crawl actually produces. A purely FIFO frontier performs breadth-first traversal and surfaces shallow pages first. A priority frontier can reach high-value pages early, which matters when a crawl may be cut short. A frontier without freshness scheduling will recrawl stale and fresh pages at the same rate, wasting requests on content that has not changed since incremental crawling last visited it.

Bright Data’s Crawl API manages frontier state, deduplication, and per-host scheduling internally, so crawls scale without requiring a custom queueing layer.