A web crawler — also called a spider or bot — is a program that systematically browses the internet by following hyperlinks from page to page, downloading content, and extracting new URLs to visit. It is the foundational component of every search engine, powering the index that makes search possible. Without crawlers, search engines would have no data to search over.

Google operates the largest web crawler in the world (Googlebot), which discovers and re-fetches pages across an index of over 400 billion documents. Bing, Yandex, Baidu, and DuckDuckGo each operate their own crawlers. Beyond search, crawlers power academic datasets like Common Crawl (which produces a free, open archive of the web containing over 3 billion pages per monthly snapshot), price comparison engines, archival services like the Internet Archive, SEO analysis tools, and machine learning training pipelines.

The scale of the modern web is staggering. Estimates put the indexed web at around 60 billion pages, with the deep web (content behind forms, logins, and dynamic generation) being orders of magnitude larger. A production crawler must handle pages in hundreds of languages, content types ranging from static HTML to JavaScript-rendered single-page applications, and an adversarial environment where some sites actively try to trap or block crawlers.

Understanding these numbers is critical because they drive every architectural decision. A toy crawler on your laptop can fetch a few pages per second using a single thread. A production crawler serving a search engine must fetch thousands of pages per second, coordinate across hundreds of machines, avoid re-fetching billions of already-seen URLs, and do all of this while being a polite guest on every web server it visits.

Every system design interview should begin with clarifying requirements and estimating scale. For a web crawler, the functional and non-functional requirements are well-defined, but the scale numbers drive radically different architectural choices.

These numbers reveal important constraints. At 11,574 pages per second with a 1-day cycle, a single machine doing 50 fetches per second would need 230+ machines in the fetcher pool alone. The bandwidth requirement of 9.3 Gbps requires careful network planning. And 100 TB of raw HTML per cycle means storage costs add up quickly, making compression and deduplication essential.

These estimates form the skeleton of your design. Every subsequent architectural decision — how many fetcher workers, how to partition the URL frontier, whether to use Bloom filters or a database for deduplication — follows from these numbers.

The URL frontier is the heart of a web crawler. It is the data structure that holds all URLs that have been discovered but not yet fetched. The frontier determines what the crawler visits next, making it the single most important component for crawl efficiency, politeness, and coverage. A naive frontier (a simple FIFO queue) works for toy crawlers but fails catastrophically at scale.

Seed URLs bootstrap the crawl. For a general-purpose search engine crawler, seeds typically include the homepages of major websites (top 10,000 by traffic), URLs submitted by webmasters through tools like Google Search Console, and URLs discovered from sitemaps. For a focused crawler (e.g., crawling only news sites), seeds are a curated list of target domains.

A production URL frontier has two layers: a front queue system that handles prioritization, and a back queue system that handles politeness. This two-layer design, described in the Mercator crawler paper (Heydon and Najork, 1999), remains the standard architecture used by modern crawlers.

graph TB
    subgraph "URL Frontier (Mercator Design)"
        direction TB
        NEW["Newly Discovered URLs"] --> NORM["URL Normalizer"]
        NORM --> DEDUP["URL Dedup Filter<br/>(Bloom Filter)"]
        DEDUP -->|"New URL"| PRIO["Priority Assigner<br/>(PageRank, freshness,<br/>domain authority)"]
        DEDUP -->|"Already seen"| DROP["Drop"]

        subgraph "Front Queues (Priority)"
            PRIO --> F1["Queue 1<br/>HIGH priority"]
            PRIO --> F2["Queue 2<br/>MEDIUM priority"]
            PRIO --> F3["Queue 3<br/>LOW priority"]
        end

        subgraph "Back Queues (Politeness)"
            F1 --> ROUTER["Back-Queue Router<br/>(hash domain → queue)"]
            F2 --> ROUTER
            F3 --> ROUTER
            ROUTER --> B1["Queue: example.com<br/>next_fetch: 10:00:05"]
            ROUTER --> B2["Queue: news.org<br/>next_fetch: 10:00:02"]
            ROUTER --> B3["Queue: shop.io<br/>next_fetch: 10:00:08"]
        end

        B1 --> FETCH["Fetcher Workers"]
        B2 --> FETCH
        B3 --> FETCH
    end

    style PRIO fill:#3b82f6,stroke:#333,color:#fff
    style ROUTER fill:#f59e0b,stroke:#333,color:#fff
    style FETCH fill:#10b981,stroke:#333,color:#fff
    style DROP fill:#ef4444,stroke:#333,color:#fff

The fetch cycle is where the crawler actually retrieves content from the web. Each cycle consists of three stages: DNS resolution to find the server IP address, HTTP request to download the page, and HTML parsing to extract content and new URLs. Optimizing each stage is critical because the fetch cycle runs billions of times.

HTML parsing is the next major bottleneck after network I/O. A production crawler must handle malformed HTML gracefully because the real web is full of broken markup — unclosed tags, invalid nesting, mixed encodings, and embedded scripts. Libraries like jsoup (Java), BeautifulSoup/lxml (Python), and Golang net/html are designed for this forgiving parsing. Link extraction must handle both absolute and relative URLs, resolve base tags, handle meta refresh redirects, and recognize link-like patterns in JavaScript strings.

The web is full of duplicate content. The same page may be reachable through multiple URLs (with and without www, with different query parameter orderings, with tracking parameters appended). Different URLs may serve identical or near-identical content (printer-friendly versions, paginated views, A/B test variants). Without robust deduplication, a crawler wastes bandwidth, storage, and crawl budget re-fetching content it already has.

Deduplication happens at two levels: URL-level (have we already seen this exact URL?) and content-level (does this page contain the same content as another page we have already fetched?). Both are essential, and they use different techniques.

graph LR
    subgraph "URL Dedup Pipeline"
        URL["Extracted URL"] --> NORM2["Normalize<br/>(lowercase, strip params,<br/>remove fragment)"]
        NORM2 --> HASH["Compute Hash<br/>(MurmurHash3 128-bit)"]
        HASH --> BF{"Bloom Filter<br/>Check"}
        BF -->|"Probably new"| ADD["Add to Frontier<br/>+ Insert into Bloom Filter"]
        BF -->|"Definitely seen"| SKIP["Skip URL"]
    end

    subgraph "Content Dedup Pipeline"
        PAGE["Fetched Page HTML"] --> EXTRACT["Extract Text<br/>(strip tags, scripts, styles)"]
        EXTRACT --> SH["Compute SimHash<br/>(64-bit fingerprint)"]
        SH --> COMPARE{"Hamming Distance<br/>vs Existing<br/>Fingerprints"}
        COMPARE -->|"Distance > 3"| UNIQUE["Unique Content<br/>→ Store"]
        COMPARE -->|"Distance ≤ 3"| NEARDUP["Near-Duplicate<br/>→ Mark + Skip"]
    end

    style BF fill:#3b82f6,stroke:#333,color:#fff
    style COMPARE fill:#f59e0b,stroke:#333,color:#fff
    style SKIP fill:#ef4444,stroke:#333,color:#fff
    style NEARDUP fill:#ef4444,stroke:#333,color:#fff
    style UNIQUE fill:#10b981,stroke:#333,color:#fff

Content-level deduplication detects when different URLs serve the same or nearly identical content. This is surprisingly common: product pages with different sorting orders, forum threads with different pagination, and syndicated articles appearing on multiple sites. Two key algorithms dominate this space: SimHash for near-duplicate detection and MinHash for document similarity estimation.

A single machine cannot crawl the web at scale. With targets of thousands of pages per second and a frontier containing billions of URLs, the crawler must be distributed across many machines. The key challenge is partitioning the work so that each machine operates independently as much as possible, while still coordinating enough to avoid duplicate work and maintain global politeness.

The standard distributed crawler architecture assigns each worker a partition of the URL space. The most common partitioning strategy is by domain: all URLs belonging to a given domain are assigned to the same worker. This has a critical advantage — per-domain politeness (rate limiting, robots.txt caching) is handled locally within each worker without any cross-worker coordination.

graph TB
    subgraph "Distributed Crawler Architecture"
        SEEDS["Seed URLs"] --> DIST["URL Distributor<br/>(Consistent Hash<br/>by domain)"]

        subgraph "Worker Node 1"
            DIST -->|"domains A-F"| F1["URL Frontier 1"]
            F1 --> FET1["Fetcher Pool<br/>(50 async connections)"]
            FET1 --> PARSE1["Parser"]
            PARSE1 -->|"New URLs"| DEDUP1["Dedup Filter"]
        end

        subgraph "Worker Node 2"
            DIST -->|"domains G-N"| F2["URL Frontier 2"]
            F2 --> FET2["Fetcher Pool<br/>(50 async connections)"]
            FET2 --> PARSE2["Parser"]
            PARSE2 -->|"New URLs"| DEDUP2["Dedup Filter"]
        end

        subgraph "Worker Node 3"
            DIST -->|"domains O-Z"| F3["URL Frontier 3"]
            F3 --> FET3["Fetcher Pool<br/>(50 async connections)"]
            FET3 --> PARSE3["Parser"]
            PARSE3 -->|"New URLs"| DEDUP3["Dedup Filter"]
        end

        DEDUP1 -->|"Route by domain"| DIST
        DEDUP2 -->|"Route by domain"| DIST
        DEDUP3 -->|"Route by domain"| DIST

        FET1 --> STORE["Content Store<br/>(S3 / HDFS)"]
        FET2 --> STORE
        FET3 --> STORE

        PARSE1 --> META["Metadata DB<br/>(URL, timestamp,<br/>fingerprint, status)"]
        PARSE2 --> META
        PARSE3 --> META
    end

    style DIST fill:#3b82f6,stroke:#333,color:#fff
    style STORE fill:#10b981,stroke:#333,color:#fff
    style META fill:#f59e0b,stroke:#333,color:#fff

Inter-worker communication is the other key design decision. When a parser on Worker 1 extracts a URL belonging to a domain owned by Worker 2, it must route that URL to Worker 2. The two common approaches are: (1) a shared message queue (Kafka or RabbitMQ) where each worker consumes from a topic partition mapped to its domain set, or (2) direct RPC where workers send URLs to the owning worker via gRPC. The message queue approach is preferred because it decouples workers and handles backpressure naturally.

Politeness is not optional — it is a fundamental requirement of any responsible web crawler. An impolite crawler can overwhelm small web servers, trigger DDoS protection systems, get its IP addresses blocked, and potentially violate computer fraud and abuse laws. Every production crawler must implement strict politeness policies at multiple levels.

The robots.txt standard (formally the Robots Exclusion Protocol, standardized as RFC 9309 in 2022) is the primary mechanism by which webmasters communicate their crawling preferences. Every crawler must fetch, parse, and obey robots.txt before crawling any page on a domain.

Legal considerations vary by jurisdiction. In the United States, the Computer Fraud and Abuse Act (CFAA) prohibits unauthorized access to computer systems. Courts have generally held that crawling publicly accessible pages is not a CFAA violation, but crawling pages behind login walls or ignoring explicit robots.txt restrictions may be. The European Union GDPR imposes restrictions on collecting personal data from web pages. The EU Database Directive protects databases, which can include structured website content.

The web is not a friendly graph. It contains intentional and unintentional structures designed to trap crawlers into infinite loops, consuming resources while providing no useful content. A production crawler must detect and escape these traps to protect its crawl budget.

Spider traps are the most common adversarial pattern. A spider trap is a set of URLs that generates an effectively infinite number of pages. The most notorious example is a dynamic calendar that generates pages for every day into the infinite future: /calendar/2024/01/01, /calendar/2024/01/02, ..., /calendar/9999/12/31. Each page links to the next day, creating an endless chain that a naive BFS crawler will follow forever.

Loop detection requires tracking the crawl path. Maintain a per-domain visit history and detect when the crawler is revisiting the same content (as measured by content fingerprints) through different URLs. If more than 10 consecutive fetches from the same domain produce near-identical SimHash fingerprints, flag the domain as a potential trap and stop crawling beyond a conservative page limit.

A crawler that only visits each page once produces a stale index. The web is constantly changing: news sites update every minute, e-commerce prices change daily, and even static blogs get occasional edits. Recrawl scheduling determines how often to revisit previously crawled pages to maintain freshness — a critical quality metric for search engines.

The core challenge is that crawl budget is finite. If you can fetch 10,000 pages per second, you must allocate those fetches between discovering new pages and recrawling known pages. Spending too much budget on recrawling means slow discovery of new content; spending too little means your index becomes stale. The optimal allocation depends on page change rates, which vary enormously across the web.

Adaptive recrawl scheduling uses historical change data to set per-page recrawl intervals. The idea is simple: if a page changed the last 5 times you checked, check it more often; if it has not changed in the last 10 checks, check it less often. This is a bandit-style optimization problem.

Priority scoring for recrawl combines multiple signals into a single score that determines which pages get recrawl budget. The formula typically looks like: recrawl_priority = page_importance_score * change_probability * (time_since_last_crawl / expected_interval). Pages that are important, likely to have changed, and overdue for a check receive the highest priority.