Why AI agents break
- Agents stop at the first error page.
- Built-in fetch tools can return error pages as if they were content.
- Retrying from the same address only makes it worse.
curl https://api.doublecrawl.com/v1/request \
-H "Authorization: Bearer $DOUBLECRAWL_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://listings.example/austin", "country": "us"}'Or ask your agent: "Use DoubleCrawl to open https://listings.example/austin". Read the docs →
Ask for a page, get the real page
One MCP tool
Add the server once. The agent can call fetch_page when a plain fetch fails. You can also ask for it by name.
Honest results
We retry error pages. If a page still fails, the result says so and costs 0.
Costs you can see
Only delivered pages are billed, so retries never burn credits.
Questions
Which agents does DoubleCrawl work with?
Any MCP client with Streamable HTTP, including Claude, ChatGPT, Cursor, VS Code, Windsurf, Gemini CLI, Codex and Zed, plus SDKs such as LangChain, OpenAI Agents SDK, Vercel AI SDK and CrewAI.
Does my agent need an API key?
MCP clients sign you in on first use. SDKs use an API key from the dashboard.
What does it cost?
5,000 free credits to start, then from $29 a month. Failed pages cost 0 credits.