Why custom tool glue fails

Agents need real web tools, not custom Python wrappers

Every team builds the same glue: a Python tool per scrape source, a half-broken session store, a homemade orchestrator that loses context after one call. Three weeks later the agent works once, breaks twice, and nobody remembers which proxy expired. We ship the four pieces every agent-loop needs out of the box — over MCP, with one HTTP endpoint.

If your agent has ever timed out on step three of a multi-step crawl — this is for you.

A custom Python tool per source

The problem. Each new agent capability is another @tool decorator: one for Amazon, one for eBay, one for the company wiki. Schema drift, version skew, and a tool registry that nobody owns. Onboarding a new model means re-implementing the wrappers from scratch.

How Yozh Scraper fixes it. One HTTP MCP endpoint at localhost:8000/mcp auto-exposes every capability — scrape_page, scrape_batch, crawl, sessions, presets. Add one line to Claude / Cursor / Cline config and the model discovers tools at runtime. Same endpoint, every client.

One MCP endpoint
Auto-discovery
Zero glue code

Stateless tool calls lose login context

The problem. Tool calls are stateless — every call starts a new browser, drops cookies, fails the second 2FA step. Agents stuck in a re-login loop chew through tokens and give up on the third retry. Manual cookie storage is brittle and leaks across runs.

How Yozh Scraper fixes it. Sessions API with a sticky session_id the agent passes between calls. Declarative login DSL (goto / fill / click / wait) or paste exported cookies once — storageState persists 24h. Every subsequent tool call reuses the session, no re-auth.

Sticky session_id
Declarative login
24h TTL

Manual auth and proxy plumbing in agent code

The problem. Every agent ends up handling API keys, proxy rotation, country codes, retries — config that has nothing to do with the model's job. Secrets leak into prompts, the agent makes "creative" decisions about retries, and proxy budgets explode in a week.

How Yozh Scraper fixes it. CYBERYOZH_API_KEY in the server's .env — agent code never sees credentials. Pick proxy: {country: "us", type: "mobile"} in the tool call, Yozh handles rotation, fingerprint, and retries server-side. Agent stays focused on reasoning.

Server-side auth
Proxy declarative
No secrets in prompts

Multi-step crawls inside an agent loop

The problem. "Find every product page on this site and pull prices" is one user intent — but in agent code it's 200 tool calls, a queue, a dedup layer, and timeout babysitting. The model wastes a context window on URL bookkeeping it should never see.

How Yozh Scraper fixes it. Yozh Crawler walks the site server-side, streams discovered URLs over SSE, and chains into the scraper on each match. The agent fires crawl_and_scrape once, gets a single stream of structured results. Server keeps the queue, the model keeps its context.

SSE streaming
Server-side queue
One tool call

How it works

From `docker compose up` to an agent calling tools — in 3 steps

One MCP endpoint, one config line in your client, zero glue code. The same flow works for Claude Desktop, Cursor, Cline, or your own agent loop.

1 Spin up the MCP endpoint

Clone the repo and docker compose up. The MCP server is live at localhost:8000/mcp — scraper, crawler, sessions, and presets all auto-exposed as tools.

# one-time setup
git clone 
  github.com/CyberYozh-data/yozh-scraper
cd yozh-scraper
docker compose up -d

2 Wire it into your client

One line in claude_desktop_config.json (or Cursor / Cline equivalent). Restart the client — tools are auto-discovered, no per-source wrappers.

{
  "mcpServers": {
    "yozh": {
      "type": "http",
      "url": "http://localhost:8000/mcp"
    }
  }
}

3 Let the agent call tools

The model picks the tool, fills in parameters, and gets structured JSON back. Sessions persist across calls. Multi-step crawls stream over SSE.

// claude calls scrape_page autonomously
{
  "tool":   "scrape_page",
  "input":  {"url": "…",
              "preset": "amazon_product"},
  "result": { /* structured JSON */ }
}

Live example

What you get back — one shape, every marketplace

Pick a preset to see what the response looks like. Field set varies per source, but the request shape is identical.

Recipes

How agents wire it up — in 10 lines each

Three concrete agent setups, ready to paste. From Claude Desktop one-liners to a full LangChain loop — same MCP endpoint behind all of them.

1 Claude Desktop research agent

Wire Yozh into Claude Desktop once. Ask the model to research a topic — it picks scrape_page / crawl autonomously and synthesizes the answer.

# claude_desktop_config.json (~/Library/...)
{
  "mcpServers": {
    "yozh": {
      "type": "http",
      "url":  "http://localhost:8000/mcp"
    }
  }
}

# restart Claude · prompt:
# "compare ssd prices on amazon vs walmart"
# → claude calls scrape_page on each PDP

HTTP transport auto tool-discovery no glue code

2 Cursor copilot with web tools

Add the same MCP server to Cursor — your IDE agent can now pull live docs, check competitor pricing, or scrape a Stack Overflow answer mid-prompt.

# .cursor/mcp.json (project root)
{
  "mcpServers": {
    "yozh": {
      "type": "http",
      "url":  "http://localhost:8000/mcp"
    }
  }
}

# prompt in Cmd+L composer:
# "fetch the latest pricing from acme.com/plans
#  and update PricingTable.tsx"

per-project config same MCP endpoint tool calls in chat

3 LangChain / Anthropic SDK loop

Plain Python agent — Anthropic SDK with mcp_servers list. One HTTP URL, model gets every Yozh tool. Works the same with LangChain's MCP adapter.

# anthropic SDK · plain python
client = Anthropic()
resp   = client.messages.create(
  model="claude-sonnet-4-5",
  mcp_servers=[{
    "type": "url",
    "url":  "https://your-host/mcp"
  }],
  messages=[{
    "role": "user",
    "content": "crawl docs.x.com, summarize"
  }],
)

SSE streaming multi-tool autopilot your orchestrator

FAQ

Common questions about MCP and agent integration

Which MCP clients are supported?

Anything that speaks the Model Context Protocol HTTP transport: Claude Desktop, Cursor, Cline, Continue, Zed, n8n MCP node, LangChain MCP adapter, and the Anthropic SDK's mcp_servers param. The endpoint at localhost:8000/mcp is one URL — you point any of them at it. No per-client wrappers or stdio shims to maintain.

How are tools registered with the agent?

Auto-discovery over MCP. When the client connects to /mcp, the server lists every available tool with its JSON schema: scrape_page, scrape_batch, crawl, create_session, list_presets, generate_preset. The model sees them as native tools at runtime — no decorators, no manual schema. New capabilities in upstream yozh-scraper appear automatically after the next pull.

Can my agent stream a multi-step crawl?

Yes — Yozh Crawler walks the site server-side and streams discovered URLs over SSE. The agent fires crawl once, gets a single event stream of structured results as they arrive. Each crawl supports scope (same-domain / same-host / allow-list), per-domain RPS, and optional chaining into the scraper so each found URL is parsed before it hits the stream. Cancel with DELETE /scrape/{id} — soft stop lets in-flight pages finish; force stop is a second call.

How do sessions persist across tool calls?

Sessions API. Create a session once with declarative login DSL (goto / fill / click / wait) or by pasting exported cookies. Yozh stores cookies + Playwright storageState server-side and gives you a sticky session_id. Every subsequent tool call that passes that ID reuses the same browser state — including 2FA-protected portals, LinkedIn, partner admins. TTL is 24h by default, refreshable. The agent never touches credentials; it just forwards the session ID it got back.

How do I extend with custom tools?

Three options. (1) Custom preset — drop a JSON file in src/presets/custom/ or call POST /api/v1/presets/generate with a sample URL and let an LLM infer the schema. The preset shows up as a parameter on the existing scrape_page tool. (2) Custom MCP tool — register your own handler in src/mcp/tools/, it joins the auto-discovered list. (3) Fork — repo is MIT-licensed, the MCP server lives in one Python module (src/mcp/server.py) and is straightforward to extend.

Tools your agent can actually call

Agents need real web tools, not custom Python wrappers

A custom Python tool per source

Stateless tool calls lose login context

Manual auth and proxy plumbing in agent code

Multi-step crawls inside an agent loop

From `docker compose up` to an agent calling tools — in 3 steps

What you get back — one shape, every marketplace

How agents wire it up — in 10 lines each

Common questions about MCP and agent integration

Ready to give your agent real tools?

Loved by data teams and AI builders

Tools your agent can actually call

Agents need real web tools, not custom Python wrappers

A custom Python tool per source

Stateless tool calls lose login context

Manual auth and proxy plumbing in agent code

Multi-step crawls inside an agent loop

From docker compose up to an agent calling tools — in 3 steps

What you get back — one shape, every marketplace

How agents wire it up — in 10 lines each

Common questions about MCP and agent integration

Ready to give your agent real tools?

Loved by data teams and AI builders

From `docker compose up` to an agent calling tools — in 3 steps