One URL in, clean structured data out

Scrape

Yozh Scraper renders any URL in a real Playwright browser and returns exactly what you ask for — extracted fields, raw HTML, or a full-page screenshot. Every scrape is an async job: submit a URL, poll the job, fetch the result. Proxies, stealth, presets, and authenticated sessions are all first-class.

• Real browser render — Playwright renders JS-built pages; flip render off for static HTML.
• Structured extraction — CSS or XPath field rules return a clean data object, far cheaper than downloading and parsing raw HTML yourself.
• Built-in proxies — CyberYozh residential / mobile LTE / datacenter, with GEO targeting and no pool-id hunting.
• Stealth by default — playwright-stealth patches (navigator.webdriver, WebGL / Canvas fingerprint, chrome runtime) to reduce bot detection.
• Presets — scrape Amazon / Google / eBay / Walmart / YouTube / LinkedIn by name, with optional LLM self-heal.
• Sessions — server-managed authenticated sessions for logged-in targets, reused across scrapes.

Quickstart

The scraper comes up from the root docker-compose.yml (alongside the crawler):

cp .env.example .env          # set CYBERYOZH_API_KEY if using proxies
docker compose up --build
# scraper → http://localhost:8000
# crawler → http://localhost:8001

Verify it's up:

curl http://localhost:8000/api/v1/health
# {"status":"ok","workers":2}

Basic usage

Every scrape endpoint creates a background job and returns a job_id. Poll the job, then fetch its results.

curl -X POST http://localhost:8000/api/v1/scrape/page \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com", "proxy_type": "none" }'
# → { "job_id": "req_abc123" }

curl http://localhost:8000/api/v1/scrape/req_abc123          # status
curl http://localhost:8000/api/v1/scrape/req_abc123/results  # results

A finished result carries metadata plus whatever you requested (data, raw_html, screenshot_base64):

{
  "job_id": "req_abc123",
  "status": "done",
  "total": 1, "done": 1,
  "results": [
    {
      "request_id": "req_abc123",
      "took_ms": 1234,
      "meta": { "url": "https://example.com", "final_url": "https://example.com/",
                "status_code": 200, "device": "desktop", "proxy_type": "none", "retries": 0 },
      "data": null, "raw_html": null, "screenshot_base64": null, "warnings": []
    }
  ]
}

Extract data

Pass an extract rule and the response includes a data object keyed by your field names — much cheaper than downloading raw_html and parsing it yourself. Rules are css or xpath.

curl -X POST http://localhost:8000/api/v1/scrape/page \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "extract": {
      "type": "css",
      "fields": {
        "title": { "selector": "h1", "attr": "text", "required": true }
      }
    }
  }'
# result → { "data": { "title": "Example Domain" } }

Each field is a rule:

Use "type": "xpath" with XPath selectors (e.g. //h1) for the same shape.

Screenshots & raw HTML

Set screenshot: true for a full-page PNG (base64 in screenshot_base64), or raw_html: true to get the full post-render HTML in raw_html.

curl -X POST http://localhost:8000/api/v1/scrape/page \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com", "screenshot": true, "raw_html": true }'

Proxies

For reliable scraping, proxies are essential — most modern sites block direct requests. Yozh integrates with the CyberYozh Proxy Service; set proxy_type on any request.

Target a location with proxy_geo (country_code / region / city). Discover what you've purchased without hunting for pool ids:

curl "http://localhost:8000/api/v1/proxies/available?proxy_type=res_rotating"
curl "http://localhost:8000/api/v1/proxies/countries"

Presets

A preset bundles a request profile + URL template + parsing recipe, so you scrape a site by name instead of hand-assembling a request. Built-ins ship for Amazon, Google, eBay, Walmart, YouTube and LinkedIn; you can also build your own (deterministic CSS/XPath or AI-generated).

curl -X POST http://localhost:8000/api/v1/scrape/preset/page \
  -H "Content-Type: application/json" \
  -d '{
    "source": "amazon_product",
    "preset_params": { "asin": "B08N5WRWNW" },
    "locale": "us",
    "llm": { "model": "openai/gpt-5.4-mini" }
  }'
# → { "job_id": "..." }  then GET /api/v1/scrape/<job_id>/results

Manage presets via GET /api/v1/presets, GET /api/v1/presets/{name}, and POST /api/v1/presets (user preset names must start with user_).

Sessions

Server-managed authenticated sessions: create one, log in once, then pass its session_id to any scrape so pages are fetched with the stored cookies + storage state. Required for logged-in targets such as the linkedin_profile preset.

1. POST /api/v1/sessions — create. Pins device / proxy_type / proxy_geo and a TTL. Returns { session_id, expires_at }.
2. POST /api/v1/sessions/{id}/login — replay a declarative login script with creds in the body: { script, creds }.
3. POST /api/v1/scrape/page with session_id set — scrape authenticated.
4. DELETE /api/v1/sessions/{id} — clean up.

# 1. create  →  {"session_id":"sess_...","expires_at":...}
curl -X POST http://localhost:8000/api/v1/sessions \
  -H "Content-Type: application/json" \
  -d '{ "device": "desktop", "proxy_type": "res_rotating" }'

# 2. log in (declarative DSL + creds)
curl -X POST http://localhost:8000/api/v1/sessions/sess_.../login \
  -H "Content-Type: application/json" \
  -d '{ "script": { "steps": [ {"op":"goto","url":"https://site/login"} ] },
        "creds": { "username": "...", "password": "..." } }'

# 3. scrape with the session
curl -X POST http://localhost:8000/api/v1/scrape/page \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://site/secure", "session_id": "sess_...", "raw_html": true }'

Batch scraping

Submit many pages in one job with POST /api/v1/scrape/pages — each entry is a full scrape request. Poll status and fetch results through the same job endpoints.

curl -X POST http://localhost:8000/api/v1/scrape/pages \
  -H "Content-Type: application/json" \
  -d '{
    "pages": [
      { "url": "https://example.com", "proxy_type": "none" },
      { "url": "https://example.org", "proxy_type": "none" }
    ]
  }'

MCP

The scraper mounts a Model Context Protocol endpoint at /mcp (Streamable HTTP). Point Claude or Cursor at it and the tools appear automatically:

• run_scrape_page
• run_scrape_pages
• get_job_status
• get_job_result
• cancel_scrape_job
• health

"yozh-scraper": {
  "type": "http",
  "url": "http://localhost:8000/mcp"
}

Then just ask: "Scrape https://example.com and tell me what's on the page." The same endpoint works from a LangChain agent or an n8n MCP Client Tool node.

Configuration reference

Request — ScrapeRequest (selected)

API reference

Important details