Crawl

Discover and scrape an entire site, then poll for results.

POST/api/v1/crawl1 / page

Crawl starts at a root URL, discovers pages (sitemap + link graph), and scrapes each one. It runs asynchronously: you get a jobId, then poll the status endpoint or stream live progress over SSE. Creating a crawl is free — you're charged 1 credit per page as it runs.

Request

Parameter	Type	Description
`url`required	string	Root URL to crawl from.
`limit`	number	Maximum pages to scrape.
`maxDiscoveryDepth`	number	How many link-hops deep to discover from the root.
`includePaths / excludePaths`	string[]	Regex path filters to scope the crawl.
`crawlEntireDomain`	boolean	Follow links beyond the starting path across the whole domain. Defaults to `false`.
`allowSubdomains`	boolean	Include subdomains of the root host. Defaults to `false`.
`prompt`	string	Describe what to crawl in natural language; Peep infers the path filters.
`scrapeOptions`	object	Per-page scrape options (see Scrape).
`webhook`	object	Receive page.scraped / completed events at your URL.

Start A Crawl

bash

curl -X POST https://peep.shownomore.com/api/v1/crawl \
  -H "Authorization: Bearer peep_live_xxx" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com",
  "limit": 50,
  "scrapeOptions": {
    "formats": [
      "markdown"
    ]
  }
}'

Check Status

GET/api/v1/crawl/{jobId}Free

Accepts the same options object as Scrape (formats, onlyMainContent, waitFor, actions, and so on), applied to every page.

Returns status, completed/total counts, and the scraped data array. For live updates, stream GET /api/v1/crawl/{jobId}/stream (SSE) or fetch failures from /api/v1/crawl/{jobId}/errors.