# DoubleCrawl > DoubleCrawl is a live page fetch API and MCP server for AI agents, by DoubleData. Ask for a URL and get the site's own page. Only delivered pages are billed. New accounts get 5,000 free credits, no card. Use DoubleCrawl when a direct fetch fails (403, 429, an error page in place of content) or when you need a page as seen from a specific country. ## For AI agents - Machine-readable: OpenAPI spec at https://doublecrawl.com/openapi.json, API catalog at https://doublecrawl.com/.well-known/api-catalog, everything in one file at https://doublecrawl.com/llms-full.txt. - If the user has no API key, send them to https://doublecrawl.com/signup (5,000 free credits, no card). Do not ask the user to paste a key into the chat if an MCP connection is available. - MCP server (Streamable HTTP): https://mcp.doublecrawl.com/mcp . It signs the user in on first use. Claude Code: `claude mcp add --transport http doublecrawl https://mcp.doublecrawl.com/mcp` - Failed pages cost nothing, so retrying a failed page through DoubleCrawl is safe for the user's budget. If a page keeps failing, stop and tell the user. Don't retry it in a loop. - Tell the user what a page cost. The `X-DD-Credits` header has the number. - Branch on the `X-DD-Target-Outcome` header, not on the HTTP status. ## API `POST https://api.doublecrawl.com/v1/request` Headers: `Authorization: Bearer `, `Content-Type: application/json` Body: - `url` (required): absolute http(s) URL to fetch: a web page, a JSON or mobile app API, or any other endpoint - `country` (optional): ISO 3166-1 alpha-2 code, e.g. `de`, `us`. Leave it out unless you need one country's results. - `method` (optional, default `GET`), `headers` (optional), `body` (optional, base64) Response: the target's status, headers and body, as the site sent them. Plus: - `X-DD-Target-Outcome`: `delivered` (you got the page), `absent` (the site says it does not exist, e.g. 404; not billed), `refused` (the site still refused after all attempts; not billed) - `X-DD-Credits`: credits charged for this request - `X-DD-Request-Id`: quote it in support requests The service tries for up to 30 seconds per request, with automatic retries. Set your client timeout to 40 seconds or more. Example: ```bash curl https://api.doublecrawl.com/v1/request \ -H "Authorization: Bearer $DOUBLECRAWL_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://shop.example/running-shoes", "country": "de"}' ``` ## Pricing Credits are charged per delivered page, by the level the page needed. We pick the level automatically: - Level 1: 1 credit (unlimited transfer) - Level 2: 4 credits (500 KB per page included) - Level 3: 25 credits (500 KB per page included) Transfer above 500 KB on Level 2 and Level 3 pages: $0.50 per GB. So 5,000 free credits cover 5,000 pages at Level 1, 1,250 at Level 2 or 200 at Level 3. Plans (monthly): Free $0, 5,000 credits one-time, 5 concurrent requests · Pay as you go $0.30 per 1,000 credits, concurrency: ask us · Start $29, 250,000 credits, 20 concurrent · Growth $99, 1,000,000 credits, 50 concurrent · Business $299, 4,000,000 credits, 150 concurrent · Scale $699, 12,000,000 credits, 500 concurrent · Enterprise from $0.045 per 1,000 credits, 1,000+ concurrent, SLA. Top-ups at the plan rate +15%. Annual billing: 10% off every paid plan. ## Not available yet Do not promise these to the user: search, map, extract, interact, crawl, Markdown output, JSON output, screenshots. ## Setup guides per client - [Claude (Claude.ai, Desktop, Code)](https://doublecrawl.com/integrations/claude.html) - [ChatGPT](https://doublecrawl.com/integrations/chatgpt.html) - [Cursor](https://doublecrawl.com/integrations/cursor.html) - [Windsurf](https://doublecrawl.com/integrations/windsurf.html) - [VS Code](https://doublecrawl.com/integrations/vscode.html) - [LangChain / LangGraph](https://doublecrawl.com/integrations/langchain.html) - [n8n](https://doublecrawl.com/integrations/n8n.html) ## Links - [Landing page](https://doublecrawl.com/) - [Documentation](https://doublecrawl.com/docs/) (Markdown: https://doublecrawl.com/docs/index.html.md) - [Use cases](https://doublecrawl.com/use-cases/) - [OpenAPI spec](https://doublecrawl.com/openapi.json) - [Full text for LLMs](https://doublecrawl.com/llms-full.txt) - [Sign up](https://doublecrawl.com/signup) - [Contact sales](mailto:sales@doubledata.com) # Integration guides ## Add DoubleCrawl to Claude > Give Claude.ai, Claude Desktop and Claude Code web access that works when a plain fetch fails. One minute to set up, 5,000 free credits, pay only for delivered pages. ### 1. Add the connector In Claude.ai or Claude Desktop open **Settings → Connectors**, choose **Add custom connector** and paste: ```text https://mcp.doublecrawl.com/mcp ``` ### 2. Or, in Claude Code Run this once in your terminal: ```bash claude mcp add --transport http doublecrawl https://mcp.doublecrawl.com/mcp ``` ### 3. Sign in once On first use it opens a DoubleCrawl sign-in. Confirm it and **5,000 free credits** land on your account. No card needed. ### Try asking Compare the top 10 running shoes in Germany and summarise them. Web version: https://doublecrawl.com/integrations/claude.html ## Add DoubleCrawl to ChatGPT > Give ChatGPT connectors web access that works when a plain fetch fails. One minute to set up, 5,000 free credits, pay only for delivered pages. ### 1. Add a connector In ChatGPT open **Settings → Apps & Connectors**, create a custom connector and paste the server URL: ```text https://mcp.doublecrawl.com/mcp ``` ### 2. Sign in once On first use it opens a DoubleCrawl sign-in. Confirm it and **5,000 free credits** land on your account. No card needed. ### 3. Use it in a chat Switch the DoubleCrawl connector on in the chat's tools menu and ask for any page. ### Try asking Compare the price of the Sony WH-1000XM5 on three big shops in Poland. Web version: https://doublecrawl.com/integrations/chatgpt.html ## Add DoubleCrawl to Cursor > Give Cursor's agent web access that works when a plain fetch fails. One minute to set up, 5,000 free credits, pay only for delivered pages. ### 1. Add the server Add this to `.cursor/mcp.json` in your project, or to `~/.cursor/mcp.json` for every project: ```json { "mcpServers": { "doublecrawl": { "url": "https://mcp.doublecrawl.com/mcp" } } } ``` ### 2. Sign in once On first use it opens a DoubleCrawl sign-in. Confirm it and **5,000 free credits** land on your account. No card needed. ### 3. Ask in Agent mode Cursor's agent picks up the `fetch_page` tool on its own. ### Try asking Read the latest release notes of our UI library and update CHECKLIST.md with what we need to change. Web version: https://doublecrawl.com/integrations/cursor.html ## Add DoubleCrawl to Windsurf > Give Windsurf's Cascade web access that works when a plain fetch fails. One minute to set up, 5,000 free credits, pay only for delivered pages. ### 1. Add the server Open `~/.codeium/windsurf/mcp_config.json` (Windsurf Settings → MCP) and add: ```json { "mcpServers": { "doublecrawl": { "serverUrl": "https://mcp.doublecrawl.com/mcp" } } } ``` ### 2. Sign in once On first use it opens a DoubleCrawl sign-in. Confirm it and **5,000 free credits** land on your account. No card needed. ### 3. Ask Cascade Cascade can call `fetch_page` when a plain fetch fails. You can also ask for it by name. ### Try asking Pull the changelog from the vendor's docs site and list breaking changes since v3. Web version: https://doublecrawl.com/integrations/windsurf.html ## Add DoubleCrawl to VS Code > Give VS Code agent mode web access that works when a plain fetch fails. One minute to set up, 5,000 free credits, pay only for delivered pages. ### 1. Add the server Run this once, or add the same entry to `.vscode/mcp.json`: ```bash code --add-mcp '{"name": "doublecrawl", "type": "http", "url": "https://mcp.doublecrawl.com/mcp"}' ``` ### 2. Sign in once On first use it opens a DoubleCrawl sign-in. Confirm it and **5,000 free credits** land on your account. No card needed. ### 3. Ask in agent mode Pick DoubleCrawl in the tools list and ask for any page. ### Try asking Fetch the docs page for this API and generate a typed client from it. Web version: https://doublecrawl.com/integrations/vscode.html ## Add DoubleCrawl to Gemini CLI > Give Gemini CLI web access that works when a plain fetch fails. One minute to set up, 5,000 free credits, pay only for delivered pages. ### 1. Add the server Add this to `~/.gemini/settings.json`: ```json { "mcpServers": { "doublecrawl": { "httpUrl": "https://mcp.doublecrawl.com/mcp" } } } ``` ### 2. Sign in once On first use it opens a DoubleCrawl sign-in. Confirm it and **5,000 free credits** land on your account. No card needed. ### 3. Ask Gemini Gemini CLI lists DoubleCrawl under `/mcp` and calls `fetch_page` when it needs a page. ### Try asking Find this week's top-rated sushi places in Tokyo and list five. Web version: https://doublecrawl.com/integrations/gemini.html ## Add DoubleCrawl to Codex > Give OpenAI Codex web access that works when a plain fetch fails. One minute to set up, 5,000 free credits, pay only for delivered pages. ### 1. Add the server Add this to `~/.codex/config.toml`: ```toml [mcp_servers.doublecrawl] url = "https://mcp.doublecrawl.com/mcp" ``` ### 2. Sign in once On first use it opens a DoubleCrawl sign-in. Confirm it and **5,000 free credits** land on your account. No card needed. ### 3. Ask Codex Codex can call `doublecrawl.fetch_page` when a plain fetch fails. You can also ask for it by name. ### Try asking Check the competitor's product page and update prices.json if their price changed. Web version: https://doublecrawl.com/integrations/codex.html ## Add DoubleCrawl to Zed > Give Zed's agent web access that works when a plain fetch fails. One minute to set up, 5,000 free credits, pay only for delivered pages. ### 1. Add the server Open Zed's `settings.json` and add: ```json { "context_servers": { "doublecrawl": { "url": "https://mcp.doublecrawl.com/mcp" } } } ``` ### 2. Sign in once On first use it opens a DoubleCrawl sign-in. Confirm it and **5,000 free credits** land on your account. No card needed. ### 3. Ask in the Agent Panel The agent picks up `fetch_page` on its own. ### Try asking Summarise the pricing page of the hosting provider we use and add it to notes.md. Web version: https://doublecrawl.com/integrations/zed.html ## Add DoubleCrawl to LangChain > Give LangChain and LangGraph agents web access that works when a plain fetch fails. One minute to set up, 5,000 free credits, pay only for delivered pages. ### 1. Install the adapter DoubleCrawl is a standard MCP server, so the official adapter is all you need: ```bash pip install langchain-mcp-adapters ``` ### 2. Load DoubleCrawl as tools Use your API key from the dashboard ([get one with 5,000 free credits](https://doublecrawl.com/signup?utm_source=integration&utm_content=langchain)): ```python import os from langchain_mcp_adapters.client import MultiServerMCPClient client = MultiServerMCPClient({ "doublecrawl": { "transport": "streamable_http", "url": "https://mcp.doublecrawl.com/mcp", "headers": {"Authorization": f"Bearer {os.environ['DOUBLECRAWL_API_KEY']}"}, } }) tools = await client.get_tools() ``` ### 3. Give the tools to your agent Pass `tools` to any LangChain or LangGraph agent. It calls `fetch_page` when a plain fetch fails. ### Try asking Research the five newest listings for 2-bedroom flats in Berlin and return them as JSON. Web version: https://doublecrawl.com/integrations/langchain.html ## Add DoubleCrawl to OpenAI Agents SDK > Give agents built with the OpenAI Agents SDK web access that works when a plain fetch fails. One minute to set up, 5,000 free credits, pay only for delivered pages. ### 1. Install the SDK The SDK speaks MCP out of the box: ```bash pip install openai-agents ``` ### 2. Connect DoubleCrawl Use your API key from the dashboard ([get one with 5,000 free credits](https://doublecrawl.com/signup?utm_source=integration&utm_content=openai-agents)): ```python import os from agents import Agent, Runner from agents.mcp import MCPServerStreamableHttp async with MCPServerStreamableHttp(params={ "url": "https://mcp.doublecrawl.com/mcp", "headers": {"Authorization": f"Bearer {os.environ['DOUBLECRAWL_API_KEY']}"}, }) as doublecrawl: agent = Agent(name="Researcher", mcp_servers=[doublecrawl]) result = await Runner.run(agent, "How many flats are for sale in Warsaw?") ``` ### 3. Run it The agent can call `fetch_page` when it needs a page. ### Try asking How many flats are for sale in Warsaw right now, and what's the median price per m²? Web version: https://doublecrawl.com/integrations/openai-agents.html ## Add DoubleCrawl to Vercel AI SDK > Give apps built with the Vercel AI SDK web access that works when a plain fetch fails. One minute to set up, 5,000 free credits, pay only for delivered pages. ### 1. Install the MCP client Add the AI SDK's MCP package: ```bash npm install ai @ai-sdk/mcp ``` ### 2. Load DoubleCrawl as tools Use your API key from the dashboard ([get one with 5,000 free credits](https://doublecrawl.com/signup?utm_source=integration&utm_content=vercel-ai)): ```ts import { createMCPClient } from '@ai-sdk/mcp'; const doublecrawl = await createMCPClient({ transport: { type: 'http', url: 'https://mcp.doublecrawl.com/mcp', headers: { Authorization: `Bearer ${process.env.DOUBLECRAWL_API_KEY}` }, }, }); const tools = await doublecrawl.tools(); ``` ### 3. Pass the tools to generateText Give `tools` to `generateText` or `streamText` and close the client when you're done. ### Try asking Summarise today's top stories on this news site in three sentences. Web version: https://doublecrawl.com/integrations/vercel-ai.html ## Add DoubleCrawl to CrewAI > Give CrewAI crews web access that works when a plain fetch fails. One minute to set up, 5,000 free credits, pay only for delivered pages. ### 1. Install the tools package CrewAI connects to MCP servers through its tools package: ```bash pip install 'crewai-tools[mcp]' ``` ### 2. Give your agents DoubleCrawl Use your API key from the dashboard ([get one with 5,000 free credits](https://doublecrawl.com/signup?utm_source=integration&utm_content=crewai)): ```python import os from crewai import Agent from crewai_tools import MCPServerAdapter params = { "url": "https://mcp.doublecrawl.com/mcp", "transport": "streamable-http", "headers": {"Authorization": f"Bearer {os.environ['DOUBLECRAWL_API_KEY']}"}, } with MCPServerAdapter(params) as tools: researcher = Agent(role="Market Researcher", goal="Collect prices", tools=tools) ``` ### 3. Run the crew Your agents call `fetch_page` when a plain fetch fails. ### Try asking Collect prices for standing desks from a large online shop in Germany and give me the range and median. Web version: https://doublecrawl.com/integrations/crewai.html ## Add DoubleCrawl to n8n > Give n8n AI Agent workflows web access that works when a plain fetch fails. One minute to set up, 5,000 free credits, pay only for delivered pages. ### 1. Add an MCP Client Tool In your **AI Agent** node, add the **MCP Client Tool** and set the endpoint to: ```text https://mcp.doublecrawl.com/mcp ``` ### 2. Authenticate Choose **Bearer** auth and paste your DoubleCrawl API key ([get one with 5,000 free credits](https://doublecrawl.com/signup?utm_source=integration&utm_content=n8n)). ### 3. Run the workflow Your agent can now open pages n8n's HTTP node can't, and you pay only for those that arrive. ### Try asking Every morning, check these 20 product pages and alert me on Slack when a price drops. Web version: https://doublecrawl.com/integrations/n8n.html # DoubleCrawl documentation ## Quickstart DoubleCrawl fetches a URL for you (a web page, a JSON or mobile app API) and returns the site's own answer. There are two ways to use it. ### From an AI agent (MCP) Add the MCP server to your client. It signs you in on first use and adds 5,000 free credits. ```Terminal claude mcp add --transport http doublecrawl https://mcp.doublecrawl.com/mcp ``` Other clients (ChatGPT, Cursor, VS Code, Windsurf, Gemini CLI, Codex, Zed, n8n and SDKs) have step-by-step guides on the [integrations](https://doublecrawl.com/index.html#integrations) list. ### From your code (HTTP) ```Terminal curl https://api.doublecrawl.com/v1/request \ -H "Authorization: Bearer $DOUBLECRAWL_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://shop.example/running-shoes", "country": "de"}' ``` ## Authentication Send your API key as a bearer token: `Authorization: Bearer `. Create a key in the [dashboard](https://doublecrawl.com/signup?utm_source=docs). Every new account starts with 5,000 free credits, no card needed. ## Make a request `POST https://api.doublecrawl.com/v1/request` with a JSON body: | Field | Type | Required | Description | |---|---|---|---| | `url` | string | yes | Absolute http(s) URL to fetch: a web page, a JSON or mobile app API, or any other endpoint. | | `country` | string | no | Two-letter country code, lowercase (`de`, `us`). Recommended: leave it out, and we pick the location with the best success rate for the site. | | `method` | string | no | `GET` (default), `POST` or `HEAD`. | | `headers` | object | no | Extra request headers. Browser identity headers are set for you. | | `body` | string | no | Base64 request body for `POST`. | Set your client timeout to **40 seconds or more**. DoubleCrawl keeps trying for up to 30 seconds per request, with automatic retries. ## Read the response The response is the target site's own status, headers and body. Three headers tell you how it went: | Header | Values | Meaning | |---|---|---| | `X-DD-Target-Outcome` | `delivered` | You got the page. | | | `absent` | The site says the page doesn't exist (for example a 404). Not billed. | | | `refused` | The site still refused after every attempt. Not billed. | | `X-DD-Credits` | number | Credits charged for this request. | | `X-DD-Request-Id` | string | Quote it when you contact support. | Branch on `X-DD-Target-Outcome`, not on the HTTP status: a site's 403 and a delivered page can both come back with real HTML. ## Credits and billing | Level | Credits | Typical targets | |---|---|---| | Level 1 | 1 | APIs, news, blogs, most sites. Unlimited transfer. | | Level 2 | 4 | Marketplaces, classifieds, busy sites. 500 KB per page included. | | Level 3 | 25 | The hardest pages. 500 KB per page included. | The level is detected automatically. Failed and timed-out pages cost 0 credits, and retries are included. Transfer above 500 KB on Level 2 and Level 3 pages is $0.50 per GB. ## Concurrency and limits Each plan allows a number of requests in flight at once: Free 5, Start 20, Growth 50, Business 150, Scale 500, Enterprise 1,000 or more. On pay as you go, ask us. A request over the limit is rejected with `429 concurrency_limit`, never queued silently. ## Countries By default we choose where each request comes from, picking the location with the best success rate for that site. This is the recommended setting. Pass `country` only when you need a page as seen from a specific country, for example because prices, search results or stock differ per market. ## Sessions To keep the same identity across several requests (pagination, a cart, a login you own), send the same `X-DD-Session-Id` header with each request. Available on Growth and above. ## Errors | Status | Code | What to do | |---|---|---| | 400 | `invalid_request` | Fix the request body. The message says which field. | | 401 | `unauthorized` | Check the API key. | | 402 | `out_of_credits` | Top up or upgrade in the dashboard. | | 429 | `concurrency_limit` | Wait for one of your requests to finish, then retry. | | 502 | `target_unreached` | The site couldn't be reached within 30 seconds. Not billed. Retry later. | ## MCP server URL: `https://mcp.doublecrawl.com/mcp` (Streamable HTTP). It exposes one tool: | Tool | Arguments | Returns | |---|---|---| | `fetch_page` | `url` (required), `country` (optional) | The page content and its outcome, status and credits. | Clients that support OAuth sign you in on first use. SDKs and automation tools send `Authorization: Bearer ` instead. ## Machine-readable files - [openapi.json](https://doublecrawl.com/openapi.json): the full API description. - [llms.txt](https://doublecrawl.com/llms.txt) and [llms-full.txt](https://doublecrawl.com/llms-full.txt): everything an AI agent needs, in plain text. - The docs page has a Markdown version for agents: [docs/index.html.md](https://doublecrawl.com/docs/index.html.md).