Quickstart
DoubleCrawl fetches a URL for you (a web page, a JSON or mobile app API) and returns the site's own answer. There are two ways to use it.
From an AI agent (MCP)
Add the MCP server to your client. It signs you in on first use and adds 5,000 free credits.
claude mcp add --transport http doublecrawl https://mcp.doublecrawl.com/mcp
Other clients (ChatGPT, Cursor, VS Code, Windsurf, Gemini CLI, Codex, Zed, n8n and SDKs) have step-by-step guides on the integrations list.
From your code (HTTP)
curl https://api.doublecrawl.com/v1/request \
-H "Authorization: Bearer $DOUBLECRAWL_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://shop.example/running-shoes", "country": "de"}'Authentication
Send your API key as a bearer token: Authorization: Bearer <key>. Create a key in the dashboard. Every new account starts with 5,000 free credits, no card needed.
Make a request
POST https://api.doublecrawl.com/v1/request with a JSON body:
| Field | Type | Required | Description |
|---|---|---|---|
url | string | yes | Absolute http(s) URL to fetch: a web page, a JSON or mobile app API, or any other endpoint. |
country | string | no | Two-letter country code, lowercase (de, us). Recommended: leave it out, and we pick the location with the best success rate for the site. |
method | string | no | GET (default), POST or HEAD. |
headers | object | no | Extra request headers. Browser identity headers are set for you. |
body | string | no | Base64 request body for POST. |
Set your client timeout to 40 seconds or more. DoubleCrawl keeps trying for up to 30 seconds per request, with automatic retries.
Read the response
The response is the target site's own status, headers and body. Three headers tell you how it went:
| Header | Values | Meaning |
|---|---|---|
X-DD-Target-Outcome | delivered | You got the page. |
absent | The site says the page doesn't exist (for example a 404). Not billed. | |
refused | The site still refused after every attempt. Not billed. | |
X-DD-Credits | number | Credits charged for this request. |
X-DD-Request-Id | string | Quote it when you contact support. |
Branch on X-DD-Target-Outcome, not on the HTTP status: a site's 403 and a delivered page can both come back with real HTML.
Credits and billing
| Level | Credits | Typical targets |
|---|---|---|
| Level 1 | 1 | APIs, news, blogs, most sites. Unlimited transfer. |
| Level 2 | 4 | Marketplaces, classifieds, busy sites. 500 KB per page included. |
| Level 3 | 50 | The hardest pages. 500 KB per page included. |
The level is detected automatically. Failed and timed-out pages cost 0 credits, and retries are included. Transfer above 500 KB on Level 2 and Level 3 pages is $0.50 per GB.
Concurrency and limits
Each plan allows a number of requests in flight at once: Free 5, Start 20, Growth 50, Business 150, Scale 500, Enterprise 1,000 or more. On pay as you go, ask us. A request over the limit is rejected with 429 concurrency_limit, never queued silently.
Countries
By default we choose where each request comes from, picking the location with the best success rate for that site. This is the recommended setting. Pass country only when you need a page as seen from a specific country, for example because prices, search results or stock differ per market.
Sessions
To keep the same identity across several requests (pagination, a cart, a login you own), send the same X-DD-Session-Id header with each request. Available on Growth and above.
Errors
| Status | Code | What to do |
|---|---|---|
| 400 | invalid_request | Fix the request body. The message says which field. |
| 401 | unauthorized | Check the API key. |
| 402 | out_of_credits | Top up or upgrade in the dashboard. |
| 429 | concurrency_limit | Wait for one of your requests to finish, then retry. |
| 502 | target_unreached | The site couldn't be reached within 30 seconds. Not billed. Retry later. |
MCP server
URL: https://mcp.doublecrawl.com/mcp (Streamable HTTP). It exposes one tool:
| Tool | Arguments | Returns |
|---|---|---|
fetch_page | url (required), country (optional) | The page content and its outcome, status and credits. |
Clients that support OAuth sign you in on first use. SDKs and automation tools send Authorization: Bearer <key> instead.
Machine-readable files
- openapi.json: the full API description.
- llms.txt and llms-full.txt: everything an AI agent needs, in plain text.
- This page has a Markdown version for agents: docs/index.html.md.