CrawlGraph: Concurrent Web Scraper & Link Indexer

CrawlGraph is a multi-threaded Python crawler that starts from one URL, crawls pages concurrently, builds a directed link graph, exports that graph as JSON, and ranks discovered pages with a from-scratch PageRank implementation.

The project is intentionally built around concurrency fundamentals: threading.Thread, queue.Queue, threading.Lock, threading.RLock, threading.Event, shared mutable state, and tests that prove the synchronization behavior under contention.

Why This Project Matters

Web crawling is mostly network I/O. A single-threaded crawler spends much of its time waiting for servers to respond. CrawlGraph uses multiple worker threads so one thread can wait on a response while others continue fetching and parsing pages.

That creates real shared-state problems:

• Two workers must not crawl the same URL twice.
• Multiple workers must not corrupt the frontier, graph, stats counters, or robots cache.
• Per-domain rate limits must remain polite even when several threads target the same host.
• The crawler must shut down cleanly when either the page cap is reached or the queue drains naturally.

Architecture

queue.Queue frontier
        |
        v
Worker-0  Worker-1  ...  Worker-N
        |
        v
shared state protected by synchronization primitives

visited set        -> threading.Lock
domain timers      -> threading.Lock
LinkGraph._adj     -> threading.RLock
CrawlStats         -> threading.Lock
robots cache       -> threading.Lock + per-domain Event
shutdown signal    -> threading.Event

File Layout

.
├── main.py
├── crawler.py
├── link_graph.py
├── robots_cache.py
├── stats.py
├── utils.py
├── requirements.txt
└── tests/
    ├── test_crawler_threading.py
    ├── test_link_graph.py
    └── test_robots_cache.py

Key Synchronization Decisions

Atomic visited check

The crawler wraps the if url not in visited: visited.add(url) sequence in _visited_lock. Without the lock, two threads can both pass the membership check before either writes to the set, causing duplicate crawls. The test suite starts 50 threads racing on the same URL and asserts that exactly one wins.

Thread-safe frontier

The crawl frontier is a queue.Queue, not a list plus a homemade lock. queue.Queue is internally synchronized and gives worker threads blocking get(timeout=...), so they sleep when no work is available instead of spinning.

Per-domain polite delay

_domain_lock protects the last-hit timestamp for each domain. Threads targeting different domains can proceed independently, while threads hitting the same domain are spaced by the configured delay.

Single-flight robots cache

RobotsCache avoids holding its lock during network I/O. The first thread for a domain becomes the fetcher, stores a threading.Event in _inflight, fetches robots.txt outside the lock, then publishes the result and wakes waiters. Other threads for the same domain wait on the event instead of launching duplicate network requests.

Reentrant graph lock

LinkGraph uses threading.RLock around its adjacency dictionary. The graph is a shared mutable object and an RLock keeps future nested graph operations from deadlocking if a locked method calls another locked method.

Clean termination

The crawler exits when the queue has no unfinished tasks or when max_pages sets the stop event. This avoids a common worker-pool bug where all workers are alive but blocked on Queue.get(), causing the main thread to wait forever after the frontier has drained.

PageRank

After crawling, CrawlGraph computes PageRank with the iterative power method:

PR(u) = (1 - d) / N + d * sum(PR(v) / out_degree(v) for v -> u)

• Default damping factor: 0.85
• Convergence threshold: 1e-6
• Dangling nodes redistribute rank uniformly
• Computation runs on a point-in-time snapshot of the graph

Setup

python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt

On macOS/Linux, activate with:

source .venv/bin/activate

Usage

Run a basic crawl:

python main.py https://docs.python.org/3/ --workers 8 --max-pages 150 --max-depth 3

Export the graph:

python main.py https://docs.python.org/3/ --workers 8 --max-pages 150 --output graph.json

Show worker activity:

python main.py https://docs.python.org/3/ -v

Run the benchmark mode:

python main.py https://docs.python.org/3/ --benchmark --workers 8 --max-pages 40

Benchmark mode crawls the same site twice: once with 1 thread and once with N threads, then prints throughput and speedup.

Benchmark Result

Measured on https://docs.python.org/3/ with 40 pages:

1 thread  : 1.12 pages/sec
8 threads : 6.41 pages/sec
Speedup   : 5.72x

Tests

python -m pytest -q

Current coverage focuses on:

• Atomic visited-set check under 50-thread contention
• Crawler shutdown when the frontier drains before max_pages
• Constructor validation for invalid crawl limits
• Exact page-cap enforcement under concurrent workers
• Thread-safe graph writes
• PageRank normalization
• Robots cache single-flight behavior under concurrent calls