Why most pipelines rot

DIY scrapers eat 90% of analytics time and ship dirty data

A "small Python script" turns into months of cookie cycling, proxy rotation, selector fixes, retry queues, and 3 a.m. pages because the source changed a class name. The data team spends its quarter rebuilding scaffolding instead of building dashboards. We ship the four pieces every team rebuilds from scratch — out of the box.

If your warehouse has more nulls than rows after a source redesign — this is for you.

Brittle DIY scrapers rot in a quarter

The problem. A weekend Python script turns into a fragile in-house engine: a queue here, a retry layer there, no observability, a CSS selector that breaks every time the source redesigns. The team owns infrastructure it never wanted to write.

How Yozh Scraper fixes it. Production-ready engine: parallel batch executor, automatic retries with exponential back-off, structured error codes, and LLM self-heal that regenerates broken selectors from the live DOM. One docker compose up replaces months of glue code.

LLM self-heal
Batch + retries
Structured errors

HTML mess across every source

The problem. Every site returns its own snowflake of nested divs, undocumented JSON blobs, and inconsistent date formats. Your loaders need a normalization step before the warehouse — and that step breaks first every time the source ships a redesign.

How Yozh Scraper fixes it. Built-in presets return identical JSON shape across sources — same keys, ISO dates, locale-normalized currencies. Custom sources? POST /api/v1/presets/generate with one example URL — an LLM infers the schema and writes selectors. Drops straight into COPY ... FROM stdin.

Same JSON shape
ISO dates
LLM-generated presets

Anti-bot fights eat the sprint

The problem. Datacenter proxies burn out in one session, CAPTCHA walls block crawls mid-job, TLS fingerprinting flags headless browsers in seconds. Data engineers end up triaging blocks instead of building dashboards. Proxy bills explode while volume stalls.

How Yozh Scraper fixes it. Native integration with CyberYozh App Proxy via CYBERYOZH_API_KEY — 5 proxy types (residential rotating/sticky, mobile 4G/5G, datacenter, ISP) across 250 country codes. Stealth Chromium build with warm fingerprint matches the proxy origin automatically. Most flows never see a CAPTCHA.

5 proxy types
250 country codes
Warm fingerprint

Crawl orchestration ad-hoc

The problem. "Walk this site, scrape every product page, watch for new URLs" is one logical job — but in code it's a queue, a dedup table, scope rules, RPS limits, two-stage cancel. Hand-rolled, it breaks under retries and silently re-scrapes the same URL.

How Yozh Scraper fixes it. Yozh Crawler walks the site server-side, dedups via per-job SHA1 hashes, streams new URLs over SSE, and chains into the scraper on each match. Two-stage cancel (soft → force). Drop the stream straight into Airflow / dbt / cron — orchestration stays in the tools you already use.

SSE streaming
Per-job dedup
Airflow / dbt / cron

How it works

From raw web page to warehouse row — in 3 steps

One endpoint to extract, one shape to load. Plug Yozh straight into your existing loader, your orchestrator, your warehouse — no middle layer needed.

1 Spin up the engine

Clone the repo and docker compose up. Scraper on :8000, crawler on :8001. No SaaS account, no API key wizard — runs on your VPC.

# one-time setup
git clone 
  github.com/CyberYozh-data/yozh-scraper
cd yozh-scraper
docker compose up -d

2 Batch-scrape a URL list

POST your URL batch (up to 200 per call). Workers run in parallel, retries and proxies handled server-side. Same JSON shape across every source.

curl 
  localhost:8000/api/v1/scrape/batch 
  -X POST 
  -d '{"urls": ["…", "…"],
       "preset": "auto",
       "concurrency": 8}'

3 Pipe straight into your warehouse

Results stream as JSON lines. Pipe through jq into COPY ... FROM stdin on Postgres, or load directly via Snowflake / BigQuery client. Zero middleware.

# postgres example
curl :8000/api/v1/scrape/batch 
  -d @urls.json 
| jq -c '.results[]' 
| psql -c "COPY raw FROM stdin"

Live example

What you get back — one shape, every marketplace

Pick a preset to see what the response looks like. Field set varies per source, but the request shape is identical.

Recipes

How data teams wire it up — in 10 lines each

Three concrete pipelines from page to warehouse, ready to paste. Yozh handles the scrape — your orchestrator handles the schedule.

1 Daily snapshot → Postgres

Cron pulls a URL list, batch-scrapes, streams JSON lines straight into a staging table. dbt picks it up downstream.

# crontab: every day at 06:00
0 6 * * * /usr/local/bin/snapshot.sh

# snapshot.sh
curl -s :8000/api/v1/scrape/batch 
  -d "@urls.json" 
| jq -c '.results[]' 
| psql -c "COPY raw_scrape FROM stdin"

# dbt build runs next in Airflow DAG
dbt build --select staging.raw_scrape+

200 URLs/batch COPY FROM stdin your cron / Airflow

2 Discovery crawl → BigQuery

Crawl a site, stream new URLs as they're discovered. Dedup against yesterday, scrape only deltas, insert rows into BigQuery as they arrive.

# stream discovered URLs over SSE
curl -N :8001/api/v1/crawl/start 
  -d '{"seed_url":"site.com",
       "scope":"same-domain",
       "max_pages":1000}' 
| jq -r 'select(.event=="found") | .url' 
| comm -13 yesterday.txt - 
| bq insert dataset.urls --source_format=NEWLINE_DELIMITED_JSON

SSE streaming per-job dedup (SHA1) delta-only inserts

3 News firehose → Kafka

Crawl + scrape news sources, publish each article as a Kafka event. Downstream consumers (dashboards, sentiment models, alerting) subscribe by topic.

# 1. continuous crawl of source list
curl -N :8001/api/v1/crawl/start 
  -d "@sources.json" 
| jq -c 'select(.event=="scraped") | .result' 
| kcat -P -t news.raw -b kafka:9092

# 2. consumers subscribe downstream
kcat -C -t news.raw | your-pipeline

scrape + publish chain Kafka producer topic per source

FAQ

Common questions about scraping into a warehouse

What sources can I scrape?

Any public page. Built-in presets cover the highest-traffic ones — marketplaces, search engines, YouTube, LinkedIn profiles (with sessions). For custom sources call POST /api/v1/presets/generate with one sample URL — an LLM infers the schema and writes selectors, you get a reusable preset back in seconds. For one-off pages, POST /scrape/page with an explicit extract: {...} schema works without any preset. Robots.txt is honored by default; respect it for the sources you don't own.

How do I get data into Snowflake / BigQuery / Postgres?

The scraper returns JSON-lines on every batch and SSE-streams on every crawl. Pipe straight into your warehouse loader: jq -c '.results[]' | psql -c "COPY raw FROM stdin" for Postgres, bq insert for BigQuery, snowsql --query "PUT ..." + COPY INTO for Snowflake, or write to S3/GCS and let your existing Snowpipe / external table pick it up. No bespoke connector to maintain — it's just JSON over HTTP.

Can I run this inside my VPC?

Yes — Yozh is self-hosted. docker compose up brings two containers (scraper + crawler) inside whatever network you give them: VPC, on-prem, air-gapped. No phone-home, no SaaS dependency. Outbound traffic goes to the targets you scrape (and optionally to the CyberYozh App Proxy if you enable it). Logs, traces, and metrics stay where you put them — Prometheus endpoint at /metrics, structured JSON logs to stdout.

How do retries, errors, and observability work?

Every request gets an exponential back-off retry budget (configurable per call). Failures return structured error codes — BLOCKED, CAPTCHA, SELECTOR_FAIL, UPSTREAM_TIMEOUT — so your loader can branch on them instead of parsing strings. When a selector breaks after a source redesign, LLM self-heal regenerates it on the fly (pass llm: {model: "..."}) and tags the response with self_heal: true for downstream auditing. Prometheus metrics for queue depth, success rate, and proxy latency ship out of the box.

How does it fit with Airflow / dbt / Temporal?

Yozh handles the scrape; your orchestrator handles the schedule. The simplest pattern: an Airflow BashOperator (or PythonOperator with requests) calls /scrape/batch, pipes results into a staging table, then chains a dbt task downstream. Crawls return an SSE stream — use a long-running task or break it into resumable chunks via the job_id. No bundled scheduler means no lock-in: use whatever your data platform already runs.

Public web data, structured for SQL