Documentation

Everything you need to fetch pages from an AI agent or your own code.

Quickstart

DoubleCrawl fetches a URL for you (a web page, a JSON or mobile app API) and returns the site's own answer. There are two ways to use it.

From an AI agent (MCP)

Add the MCP server to your client. It signs you in on first use and adds 5,000 free credits.

Terminal
claude mcp add --transport http doublecrawl https://mcp.doublecrawl.com/mcp

Other clients (ChatGPT, Cursor, VS Code, Windsurf, Gemini CLI, Codex, Zed, n8n and SDKs) have step-by-step guides on the integrations list.

From your code (HTTP)

Terminal
curl https://api.doublecrawl.com/v1/request \
  -H "Authorization: Bearer $DOUBLECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://shop.example/running-shoes", "country": "de"}'

Authentication

Send your API key as a bearer token: Authorization: Bearer <key>. Create a key in the dashboard. Every new account starts with 5,000 free credits, no card needed.

Make a request

POST https://api.doublecrawl.com/v1/request with a JSON body:

FieldTypeRequiredDescription
urlstringyesAbsolute http(s) URL to fetch: a web page, a JSON or mobile app API, or any other endpoint.
countrystringnoTwo-letter country code, lowercase (de, us). Recommended: leave it out, and we pick the location with the best success rate for the site.
methodstringnoGET (default), POST or HEAD.
headersobjectnoExtra request headers. Browser identity headers are set for you.
bodystringnoBase64 request body for POST.

Set your client timeout to 40 seconds or more. DoubleCrawl keeps trying for up to 30 seconds per request, with automatic retries.

Read the response

The response is the target site's own status, headers and body. Three headers tell you how it went:

HeaderValuesMeaning
X-DD-Target-OutcomedeliveredYou got the page.
absentThe site says the page doesn't exist (for example a 404). Not billed.
refusedThe site still refused after every attempt. Not billed.
X-DD-CreditsnumberCredits charged for this request.
X-DD-Request-IdstringQuote it when you contact support.

Branch on X-DD-Target-Outcome, not on the HTTP status: a site's 403 and a delivered page can both come back with real HTML.

Credits and billing

LevelCreditsTypical targets
Level 11APIs, news, blogs, most sites. Unlimited transfer.
Level 24Marketplaces, classifieds, busy sites. 500 KB per page included.
Level 350The hardest pages. 500 KB per page included.

The level is detected automatically. Failed and timed-out pages cost 0 credits, and retries are included. Transfer above 500 KB on Level 2 and Level 3 pages is $0.50 per GB.

Concurrency and limits

Each plan allows a number of requests in flight at once: Free 5, Start 20, Growth 50, Business 150, Scale 500, Enterprise 1,000 or more. On pay as you go, ask us. A request over the limit is rejected with 429 concurrency_limit, never queued silently.

Countries

By default we choose where each request comes from, picking the location with the best success rate for that site. This is the recommended setting. Pass country only when you need a page as seen from a specific country, for example because prices, search results or stock differ per market.

Sessions

To keep the same identity across several requests (pagination, a cart, a login you own), send the same X-DD-Session-Id header with each request. Available on Growth and above.

Errors

StatusCodeWhat to do
400invalid_requestFix the request body. The message says which field.
401unauthorizedCheck the API key.
402out_of_creditsTop up or upgrade in the dashboard.
429concurrency_limitWait for one of your requests to finish, then retry.
502target_unreachedThe site couldn't be reached within 30 seconds. Not billed. Retry later.

MCP server

URL: https://mcp.doublecrawl.com/mcp (Streamable HTTP). It exposes one tool:

ToolArgumentsReturns
fetch_pageurl (required), country (optional)The page content and its outcome, status and credits.

Clients that support OAuth sign you in on first use. SDKs and automation tools send Authorization: Bearer <key> instead.

Machine-readable files

Your agent's next page is on us.