Pay for pages, never for the limit.
Submitting is free and you pay only for the pages a crawl actually completes.
Scoped by paths and depth
Include and exclude paths, a depth limit and a page limit keep a crawl in bounds.
Seeded from the sitemap
Sitemap URLs join the queue at depth zero, so max_depth=0 crawls only what a site lists.
Webhooks as pages land
Signed, ordered, retried crawl.page events, so there is nothing to poll.
crw_9f2c1a7b- /docs/payments
- /docs/payments/checkout
- /docs/billing/subscriptions
- /docs/changelog/2019
Crawl sites with Claude
Ask for a whole site or section as markdown, or send the same request with your API key.
# claude.ai/directory/prefetch → Connect
# Sign in to Prefetch, Allow access. No key to paste.
# Claude Code on the same Claude account has it too.
# Then just ask:
Use prefetch_crawl on docs.stripe.com/payments, 25
pages, and summarise how Stripe recommends
accepting a payment.What teams build with the Crawl API
Docs into RAG
Turn a product's documentation into a clean markdown corpus your retrieval pipeline can index.
Knowledge bases
Mirror a help centre or wiki section into your own store, and refresh it on a schedule.
Agent research
Give an agent a whole site as context, instead of one page at a time.
Content migration
Lift every page of an old site out as structured content before a redesign or a re-platform.
1 credit per crawled page
Submitting a crawl is free: you pay for the pages it returns, never for the limit you set, and failed pages cost nothing. Pages with a summary or JSON extraction cost 4. Every plan unlocks every endpoint, failed requests are free, and requests pause at your limit rather than billing past it.
Free
USD
/month
1,000 credits / month
- Up to 1,000 crawled pages
- 1 concurrent request
- 30 requests per minute
- No credit card required
Starter
USD
/month
5,000 credits / month
- Up to 5,000 crawled pages
- 10 concurrent requests
- Unlimited requests per minute
- Email support
- $5 per 2,500 additional credits
The same plan unlocks every other endpoint too. Each line is the whole month of 5,000 credits spent there instead of on crawled pages. Mix them however you like.
5,000 name lookups: Company Name to Domain API
(1 credit per lookup, a company, brand or product name in, its official website's domain out)
5,000 pages: Web Scraping API
(1 credit per page, any URL to clean markdown, a summary or structured JSON, ready for an LLM)
5,000 site maps: Sitemap API
(1 credit per call, every URL a site exposes, from its sitemaps and links, without fetching a page)
5,000 screenshots: Screenshot API
(1 credit per call, any URL to a hosted image, at any viewport, full page or above the fold)
5,000 page checks: Website Monitoring API
(1 credit per page checked, pages re-checked on your schedule, with a webhook carrying only what changed)
1,000 company lookups: Company Data API
(5 credits per call, the legal entity, its size, founding year and published contacts behind a domain)
1,000 classifications: Website Classification API
(5 credits per call, the IAB category, subcategory and a readable description of any URL)
500 brand lookups: Brand API
(10 credits per call, logos, colors, fonts and social assets pulled from any domain)
500 answers: Web Research API
(10 credits per answer, a research question in, a sourced answer in your own JSON shape out)
416 enrichments: Company Enrichment API
(12 credits per call, brand, company and classification for any domain, from one page load)
Failed requests are never charged. Top-up credits are spent after your monthly credits and never expire.
Web Crawling API questions
Scale with confidence. Handle thousands of requests per second on fast, reliable infrastructure.
- Registered Users
- 300+
- API calls processed
- 40k
- Uptime
- 99.9%
Ready to get started?
Create an account instantly, or contact us to design a custom package for your business.
Up in 5 minutes
Make your first API call in under 5 minutes. No setup headaches, just copy, paste, ship.
API reference
Request schemas, response examples, everything you need to integrate.