Why generic scrapers fail

Marketplaces break one-size-fits-all scrapers inside a quarter

E-commerce platforms ship layout updates every few weeks, deploy multi-layer anti-bot, gate Buy Box and seller data behind sessions, and return a different field shape on every locale. Most teams spend 80% of their time keeping the scraper alive and 20% on the data itself. We ship the four things they end up rebuilding from scratch.

If you've ever woken up to a dashboard full of nulls after a marketplace redesign — this is for you.

Selectors break every layout shift

The problem. Generic scrapers ship a CSS or XPath path and pray. When a marketplace redesigns a PDP — Amazon does it every few weeks — your pipeline returns silent nulls in production. You find out from a downstream BI dashboard hours later.

How Yozh Scraper fixes it. Pass llm: {model: "..."} in the request and the parser asks Anthropic / OpenAI / Gemini / OpenRouter to regenerate the selector from the live DOM. Pipeline keeps running with a self_heal: true flag in the response — no on-call wake-up.

LLM self-heal
Multi-provider
Zero downtime

One IP, five requests, ban

The problem. Datacenter proxies burn out within a single session. CAPTCHA walls, TLS fingerprinting, and JavaScript challenges flag headless browsers in seconds. Most teams build a proxy rotation layer from scratch and still get blocked.

How Yozh Scraper fixes it. Native integration with CyberYozh App Proxy via CYBERYOZH_API_KEY — 5 proxy types (residential rotating / sticky, mobile 4G/5G, datacenter, ISP) across 250 country codes. Stealth Chromium build with warm fingerprint matches the proxy origin automatically.

5 proxy types
250 country codes
Warm fingerprint

Different field shape per marketplace

The problem. Amazon's price lives in one block, eBay's in another, Walmart's in a third. Cross-marketplace dashboards need a normalization layer before the first metric ships. Teams spend weeks reconciling field shapes.

How Yozh Scraper fixes it. 10 built-in presets in src/presets/builtin/ return identical JSON across Amazon, eBay, Walmart, Google Shopping. Same keys: title, price, currency, in_stock, rating. Locale-aware currency parsing — structure stays the same.

10 presets
Same JSON shape
Locale-aware currency

Login walls and dying sessions

The problem. Account-bound data — Buy Box winner, partner portals, internal admins, LinkedIn-style profile sources — requires login. Hand-rolled cookie storage breaks on re-auth. Most scrapers don't ship session management at all.

How Yozh Scraper fixes it. Sessions API with declarative login DSL (goto / fill / click / wait) or paste exported cookies. Reuse session_id across every scrape — cookies + storageState persist 24h, refresh on the fly. Required for the LinkedIn preset, supported by every other.

Declarative login
Cookie + storageState
24h TTL

How it works

From URL to structured JSON in 3 calls

No selectors to write. Pick a preset, pass parameters, parse the response. The same flow works for every supported marketplace.

1 Spin up

Clone the repo and docker compose up. Two services start: scraper on :8000, crawler on :8001.

# one-time setup
git clone \
  github.com/CyberYozh-data/yozh-scraper
cd yozh-scraper
docker compose up -d

2 Call a preset

POST a preset name plus parameters. No selectors, no anti-bot tuning — the preset bundle handles selectors, locale, and currency parsing.

curl \
  localhost:8000/api/v1/scrape/preset/page \
  -X POST \
  -d '{"source": "amazon_product",
       "preset_params": {"asin": "B0CFFXRFTH"},
       "locale": "us"}'

3 Get structured JSON

Same shape across regions. If a selector breaks after a layout change, the LLM self-heal regenerates it on the fly — pipeline keeps running.

{
  "asin":       "B0CFFXRFTH",
  "price":      189.99,
  "currency":   "USD",
  "in_stock":   true,
  "rating":     4.7
}

Live example

What you get back — one shape, every marketplace

Pick a preset to see what the response looks like. Field set varies per source, but the request shape is identical.

Recipes

How teams wire it up — in 10 lines each

Three concrete e-commerce setups, ready to paste. No scheduler is bundled — use your own cron / Airflow / Temporal.

1 Daily price snapshot

Batch your competitor ASIN list once a day, diff against yesterday, post changes to Slack. Cron does the rest.

# crontab: every day at 06:00
0 6 * * * /usr/local/bin/price-watch.sh

# price-watch.sh
curl -X POST localhost:8000/api/v1/scrape/preset/pages \
  -d "@asins.json" \
| jq '.results[] | {asin, price, in_stock}' \
> today.json

diff yesterday.json today.json | slack-cli post #pricing
mv today.json yesterday.json

200 URLs/batch ~30s on 1 worker your cron

2 New-product detection

Crawl a category page daily, dedup product URLs against yesterday. Webhook on new arrivals before competitors index them.

# start a crawl of the category
curl -X POST localhost:8001/api/v1/crawl/start \
  -d '{"seed_url":"amazon.com/s?k=ssd",
       "max_pages":300,
       "scope":"same-domain"}' \
| jq -r '.urls[]' > today_urls.txt

# urls that appeared today
comm -13 yesterday_urls.txt today_urls.txt \
| curl -X POST $WEBHOOK_URL -d @-

SSE stream per-job dedup (SHA1) scope + rate-limit

3 Logged-in source · sessions

Scrape pages behind login (LinkedIn, partner portals, internal admins). Log in once via the Sessions API, then reuse session_id in every scrape.

# 1. log in once via declarative script
SID=$(curl -X POST \
  localhost:8000/api/v1/sessions/$ID/login \
  -d "@login_script.json" \
  | jq -r .session_id)

# 2. reuse for every scrape · cookies persist 24h
curl -X POST localhost:8000/api/v1/scrape/preset/page \
  -d "{\"source\":\"linkedin_profile\",
       \"preset_params\":{\"slug\":\"satyanadella\"},
       \"session_id\":\"$SID\"}"

24h TTL cookie + storageState CAPTCHA: paste cookies

FAQ

Common questions about marketplace data

What marketplaces do you support?

10 built-in presets in src/presets/builtin/ covering the biggest data sources: amazon_product (us, uk, de, fr, jp), amazon_search (us, uk, de), ebay_search (us, uk, de), walmart_product (us), google_search (us, uk, de, fr, ru, jp), google_shopping (us, uk, de), bing_search (us, uk, de, fr, ru, jp), yandex_search (us, uk, de, fr, ru, jp), youtube_video (global), linkedin_profile (global, requires session_id). Need something else? Use POST /api/v1/presets/generate with a sample URL — an LLM infers the schema and generates selectors for you.

How stable is the schema?

Same field shape across regions and presets — amazon_product in .us, .uk, .de, .fr, .jp all return the identical keys (currency localized, structure unchanged). Optional fields are explicitly nullable, so missing values never break your parser. When a marketplace changes layout, LLM self-heal regenerates the selector on the fly (pass llm: {model: "..."} in the request) — pipeline keeps running with a self_heal: true flag in the response. Breaking changes ship only in minor versions with migration notes in CHANGELOG.md.

Can I scrape Amazon / eBay / Walmart?

Yes — all three are first-class presets, no extra config. Amazon ships across 5 regional locales (us, uk, de, fr, jp) for product PDPs and search; eBay covers us/uk/de search; Walmart .us for product pages. Each returns the same JSON shape: title, price, currency, in_stock, rating, seller, plus marketplace-specific fields where they exist (ASIN, Buy Box winner, sponsored flag). For account-bound data behind login (Prime pricing, partner portals) use the Sessions API and pass session_id with the request.

How fast can I get first data?

Roughly 5 minutes end-to-end. git clone + docker compose up -d brings the scraper online in ~30 seconds (Docker pulls the image once). First curl POST /api/v1/scrape/preset/page returns structured JSON in 1–3 seconds for cached locales, up to 5–8 seconds cold. No signup, no API key creation, no SaaS metering — you hit the endpoint on localhost:8000 the moment the container is healthy.

Do you handle proxies and anti-bot?

Yes. The scraper integrates with the CyberYozh App Proxy via CYBERYOZH_API_KEY — 5 proxy types across 250 country codes (residential rotating / sticky, mobile 4G/5G, datacenter, ISP). Anti-bot is handled by a stealth Chromium build with warm fingerprint, matching residential proxy, and human-like timing — most flows avoid CAPTCHA entirely. When a CAPTCHA does appear, the scraper returns it as a structured error rather than solving silently. Rule of thumb: residential rotating for catalog-scale scraping, mobile for aggressive anti-bot zones or account-bound tasks.

One schema. Every marketplace