# DoubleCrawl documentation

## Quickstart

DoubleCrawl fetches a URL for you (a web page, a JSON or mobile app API) and returns the site's own answer. There are two ways to use it.

### From an AI agent (MCP)

Add the MCP server to your client. It signs you in on first use and adds 5,000 free credits.

```Terminal
claude mcp add --transport http doublecrawl https://mcp.doublecrawl.com/mcp
```

Other clients (ChatGPT, Cursor, VS Code, Windsurf, Gemini CLI, Codex, Zed, n8n and SDKs) have step-by-step guides on the [integrations](https://doublecrawl.com/index.html#integrations) list.

### From your code (HTTP)

```Terminal
curl https://api.doublecrawl.com/v1/request \
  -H "Authorization: Bearer $DOUBLECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://shop.example/running-shoes", "country": "de"}'
```

## Authentication

Send your API key as a bearer token: `Authorization: Bearer <key>`. Create a key in the [dashboard](https://doublecrawl.com/signup?utm_source=docs). Every new account starts with 5,000 free credits, no card needed.

## Make a request

`POST https://api.doublecrawl.com/v1/request` with a JSON body:

| Field | Type | Required | Description |
|---|---|---|---|
| `url` | string | yes | Absolute http(s) URL to fetch: a web page, a JSON or mobile app API, or any other endpoint. |
| `country` | string | no | Two-letter country code, lowercase (`de`, `us`). Recommended: leave it out, and we pick the location with the best success rate for the site. |
| `method` | string | no | `GET` (default), `POST` or `HEAD`. |
| `headers` | object | no | Extra request headers. Browser identity headers are set for you. |
| `body` | string | no | Base64 request body for `POST`. |

Set your client timeout to **40 seconds or more**. DoubleCrawl keeps trying for up to 30 seconds per request, with automatic retries.

## Read the response

The response is the target site's own status, headers and body. Three headers tell you how it went:

| Header | Values | Meaning |
|---|---|---|
| `X-DD-Target-Outcome` | `delivered` | You got the page. |
| | `absent` | The site says the page doesn't exist (for example a 404). Not billed. |
| | `refused` | The site still refused after every attempt. Not billed. |
| `X-DD-Credits` | number | Credits charged for this request. |
| `X-DD-Request-Id` | string | Quote it when you contact support. |

Branch on `X-DD-Target-Outcome`, not on the HTTP status: a site's 403 and a delivered page can both come back with real HTML.

## Credits and billing

| Level | Credits | Typical targets |
|---|---|---|
| Level 1 | 1 | APIs, news, blogs, most sites. Unlimited transfer. |
| Level 2 | 4 | Marketplaces, classifieds, busy sites. 500 KB per page included. |
| Level 3 | 25 | The hardest pages. 500 KB per page included. |

The level is detected automatically. Failed and timed-out pages cost 0 credits, and retries are included. Transfer above 500 KB on Level 2 and Level 3 pages is $0.50 per GB.

## Concurrency and limits

Each plan allows a number of requests in flight at once: Free 5, Start 20, Growth 50, Business 150, Scale 500, Enterprise 1,000 or more. On pay as you go, ask us. A request over the limit is rejected with `429 concurrency_limit`, never queued silently.

## Countries

By default we choose where each request comes from, picking the location with the best success rate for that site. This is the recommended setting. Pass `country` only when you need a page as seen from a specific country, for example because prices, search results or stock differ per market.

## Sessions

To keep the same identity across several requests (pagination, a cart, a login you own), send the same `X-DD-Session-Id` header with each request. Available on Growth and above.

## Errors

| Status | Code | What to do |
|---|---|---|
| 400 | `invalid_request` | Fix the request body. The message says which field. |
| 401 | `unauthorized` | Check the API key. |
| 402 | `out_of_credits` | Top up or upgrade in the dashboard. |
| 429 | `concurrency_limit` | Wait for one of your requests to finish, then retry. |
| 502 | `target_unreached` | The site couldn't be reached within 30 seconds. Not billed. Retry later. |

## MCP server

URL: `https://mcp.doublecrawl.com/mcp` (Streamable HTTP). It exposes one tool:

| Tool | Arguments | Returns |
|---|---|---|
| `fetch_page` | `url` (required), `country` (optional) | The page content and its outcome, status and credits. |

Clients that support OAuth sign you in on first use. SDKs and automation tools send `Authorization: Bearer <key>` instead.

## Machine-readable files

- [openapi.json](https://doublecrawl.com/openapi.json): the full API description.
- [llms.txt](https://doublecrawl.com/llms.txt) and [llms-full.txt](https://doublecrawl.com/llms-full.txt): everything an AI agent needs, in plain text.
- This page has a Markdown version for agents: [docs/index.html.md](index.html.md).
