API Reference

Web Crawl

Crawl an entire website starting from a root URL. Returns all discovered pages as structured content. This is an async operation — submit and poll for results.

POST/v1/web/crawl•1 + 1 credit/page crawled

Pro+ Required

Parameters

Name	Type	Required	Default	Description
`url`	`string`	required	—	Root URL to start crawling from
`maxPages`	`number`	optional	10	Maximum pages to crawl

Code Examples

from supacrawlx import Client
import time

client = Client("YOUR_API_KEY")

job = client.web.crawl("https://example.com", max_pages=20)
while job.status != "completed":
    time.sleep(5)
    job = client.jobs.get(job.job_id)

for page in job.result.pages:
    print(page.url, len(page.content))

Response

JSON

// POST response (async)
{
  "jobId": "job_crawl_abc123",
  "status": "queued"
}

// Poll GET /v1/jobs/job_crawl_abc123 — completed
{
  "jobId": "job_crawl_abc123",
  "status": "completed",
  "result": {
    "pagesCount": 10,
    "pages": [
      { "url": "https://example.com/", "title": "Home", "content": "..." },
      { "url": "https://example.com/about", "title": "About", "content": "..." }
    ]
  }
}