One seed URL in, the whole site streamed out

Crawl

Yozh Crawler walks a site from a single seed URL and streams every discovered page over SSE. It owns discovery — frontier, dedup, scope, link extraction, retries, politeness — while every page fetch is delegated to Yozh Scraper over HTTP. There's exactly one Playwright stack to operate, and every scraper feature (proxy, stealth, sessions, extract rules) applies to crawled pages too.

• Scope predicates — same-domain / subdomains / all / regex, with include & exclude patterns and hard max_depth / max_pages caps.
• Fingerprint dedup — URLs are canonicalized then hashed (SHA1) so reordered queries and trailing-slash variants collapse to one visit.
• Per-domain round-robin frontier — one slow or huge host can't starve the others of workers.
• Adaptive rate limiter — a global per-domain token bucket (with jitter) shared across jobs; a 429 halves the RPS for a cool-off window.
• Session health — Crawlee-style scoring: 401/403/429 retire a session, errors raise its score, success lowers it; rotate at score ≥ 3 or 50 uses.
• SSE streaming — every job emits stats, page, page_error, and terminal done / cancelled events. Act on finds as they land.

Quickstart

The crawler is wired into the root docker-compose.yml and comes up alongside the scraper:

docker compose up --build
# scraper  → http://localhost:8000
# crawler  → http://localhost:8001

Or run it locally without Docker — point SCRAPER_URL at a running scraper:

cd yozh-crawler
pip install -r requirements.txt
cp .env.example .env          # edit SCRAPER_URL if needed
python -m uvicorn src.main:app --reload --port 8001

Basic usage

Submit a crawl with a seed_url and a scope. The call returns a job_id immediately; the crawl then runs in the background across worker tasks.

curl -X POST http://localhost:8001/api/v1/crawl \
  -H "Content-Type: application/json" \
  -d '{
    "seed_url": "https://example.com",
    "scope": {"mode": "same-domain", "max_depth": 2, "max_pages": 50,
              "per_domain_rps": 1.0, "per_domain_concurrency": 1},
    "scrape_options": {"proxy_type": "none"},
    "enable_scraping": false
  }'

The response is just the job handle:

{ "job_id": "crawl_abc123" }

From here you either stream live events or poll the job for the full record.

Discovery vs harvest — the enable_scraping toggle

The same crawl engine runs both ways. One boolean chooses between a cheap discovery map and a full-payload harvest — and each mode uses its own proxy config, so there's no cross-pollination.

Crawl scope

Scope bounds the crawl: where it's allowed to walk, and how hard it's allowed to push each host. It's the scope object on the request.

Streaming results (SSE)

Every job exposes a Server-Sent Events stream. Read it to act on pages the moment they're discovered, instead of waiting for the crawl to finish.

curl -N http://localhost:8001/api/v1/crawl/crawl_abc123/events

The stream emits five event types:

A page event in discovery mode looks like this:

{
  "event": "page",
  "url": "https://example.com/docs/intro",
  "parent_url": "https://example.com/docs",
  "depth": 2,
  "status_code": 200,
  "took_ms": 812,
  "scrape_response": null
}

Status & results

Prefer polling? The job record is queryable any time — including mid-crawl, with the pages discovered so far. /results is an alias of the job endpoint, kept for symmetry with the scraper.

curl http://localhost:8001/api/v1/crawl/crawl_abc123
curl http://localhost:8001/api/v1/crawl/crawl_abc123/results

{
  "job_id": "crawl_abc123",
  "status": "running",
  "stats": {
    "visited": 47, "queued": 12, "failed": 0,
    "dedup_skipped": 9, "out_of_scope": 31, "retries_total": 2
  },
  "pages": [
    { "url": "https://example.com/", "parent_url": null,
      "depth": 0, "status_code": 200, "took_ms": 640 }
  ]
}

Cancelling a crawl

Stop a running job with a DELETE. Soft cancel is the default and the safe choice.

# soft — stop scheduling, let in-flight requests drain
curl -X DELETE "http://localhost:8001/api/v1/crawl/crawl_abc123?hard=false"

# hard — abort in-flight asyncio tasks immediately
curl -X DELETE "http://localhost:8001/api/v1/crawl/crawl_abc123?hard=true"

MCP

fastapi-mcp is mounted at /mcp as a Streamable HTTP endpoint. Point Claude or Cursor at it and the tools appear automatically:

• health
• create_crawl
• get_crawl
• get_crawl_results
• cancel_crawl

"open-crawler": {
  "type": "http",
  "url": "http://localhost:8001/mcp"
}

Configuration reference

Request — CrawlRequest

Per-page — ScrapeOptions (selected)

Environment

API reference

Important details