Seed URL

TLDR: A seed URL is the starting web address a crawler uses to begin discovery. Every other page in a crawl is reached by following links outward from the seeds, so seed selection determines what a crawl can and cannot find.

A seed URL is the initial entry point given to a web crawler before a crawl begins. The crawler fetches the seed, extracts its links, queues them, and repeats the process outward. A crawl can start from a single seed or from thousands. Pages that cannot be reached by following links from a seed will never be visited, which makes seed selection one of the highest-leverage decisions in any crawl configuration.

How Seed URLs Work:

1. Fetch: The crawler requests the seed URL and retrieves its HTML.
2. Extract: All links on the page are pulled out through link extraction and resolved from relative to absolute form.
3. Normalize: Discovered URLs are canonicalized through URL normalization so duplicate variants collapse into a single entry.
4. Enqueue: New URLs are added to the crawl queue, known as the URL frontier.
5. Repeat: The crawler pulls the next URL from the queue and continues until the queue empties or a limit such as crawl depth is reached.

Choosing Seed URLs:

Good seeds are link-rich pages that sit close to the content you actually want. A homepage is the obvious default but often a poor choice for targeted crawls, because valuable pages may sit many hops away behind navigation. Category listings, paginated indexes, author archives, and sitemaps usually deliver faster and more complete coverage.

For an e-commerce crawl, seeding from category listing pages reaches product pages in one or two hops instead of four or five from the homepage. For a documentation site, seeding from the table of contents captures nearly every page immediately. When a site publishes a sitemap, using its URLs as seeds bypasses link discovery entirely.

Seed Strategies Compared:

Example Crawl Configuration:

{
  "seeds": [
    "https://example.com/products/electronics",
    "https://example.com/products/home-garden",
    "https://example.com/products/clothing"
  ],
  "allowed_domains": ["example.com"],
  "max_depth": 3,
  "exclude_patterns": ["/cart", "/account", "*?sort="]
}

Three category seeds replace a single homepage seed. Every product page is reachable within the depth limit, and the exclusion patterns stop the crawler from spending requests on cart, account, and sort-parameter URLs.

Common Seed Selection Mistakes:

1. Seeding only the homepage: Pushes target content beyond the depth limit and burns requests on navigation pages.
2. Seeding search result URLs: Query-parameter pages generate near-infinite variations and can trap a crawler in an endless loop.
3. Ignoring pagination: A category seed that only reaches page one misses most of the catalog unless pagination links are explicitly followed.
4. Seeding redirect chains: Consumes requests and can drop the crawler onto an unintended domain when redirects are not handled.
5. Seeding without scope rules: Without crawl scope boundaries, outbound links can pull the crawl off-domain entirely.

Seed URLs at Scale:

At small scale, seed choice affects efficiency. At large scale it determines feasibility. A crawl of millions of pages built from a single seed spends an enormous share of its budget rediscovering the same navigation structures. Distributing thousands of seeds across parallel workers removes most of that redundant fetching, provided a shared frontier prevents two workers from crawling the same URL.

Bright Data’s Crawl API accepts single or bulk seed inputs, handles normalization and deduplication automatically, and manages the proxy layer so seeded crawls complete without being blocked mid-run.