Web Scraping

Scrape, search, map, extract, crawl, and monitor any public web page. Content-bearing responses return the unified WebPage schema — clean markdown/HTML, metadata, structured extraction, and fetch provenance under data.page — while search and map return a WebPageList (data.items[]).

Base URL: /v1/web/...

Getting Started

1. Scrape a page to markdown

curl "https://www.socialcrawl.dev/v1/web/scrape?url=https://example.com" \
  -H "x-api-key: sc_your_api_key_here"

The page content arrives at data.page.content.markdown (add formats=markdown,screenshot for a screenshot URL under data.page.media).

2. Search the web

curl "https://www.socialcrawl.dev/v1/web/search?query=social%20media%20data%20api&limit=10" \
  -H "x-api-key: sc_your_api_key_here"

Web, news, and image sources merge into one data.items[] list, each row tagged with source_type.

3. Go beyond single requests

• Structured extraction — GET /v1/web/extract with a schema (JSON) or prompt returns typed fields under data.page.extraction.
• Async crawls & batch scrapes — POST /v1/web/crawl / POST /v1/web/batch-scrape / POST /v1/web/agent return a job id (data.job_id, always job_…). Every job kind reads through one unified surface: poll with GET /v1/web/jobs/{job_id} (append /errors for per-page failures), cancel with DELETE /v1/web/jobs/{job_id}, list with GET /v1/web/jobs. Credits are held up front and settled to actual usage — unused holds refund automatically.
• Change monitoring — POST /v1/web/monitors (returns data.monitor_id, wm_…) schedules checks as tight as every 5 minutes (cadence_minutes: 5-60, or whole hours up to 1440); checks bill only when they run. List, pause/resume (PATCH), delete, and read check history via the same /v1/web/monitors family.
• Interactive sessions — POST /v1/web/sessions opens a short-lived browser you can drive step-by-step with POST /v1/web/sessions/{session_id}/execute.
• Document parse — POST /v1/web/parse (multipart file upload) turns PDFs and office documents into the same data.page shape.

Endpoints

22 endpoints available.

Notes

• Method-aware surface: the sync data endpoints (scrape, search, map, extract) and all reads are GET; async jobs, monitors, sessions, and parse use POST/PATCH/DELETE as listed above
• Authentication via x-api-key header
• Content responses follow the unified WebPage schema; job/monitor/session resources return envelope-wrapped status objects with cursor pagination on lists
• Variable-cost work (crawl pages, PDF parsing, enhanced proxy, agent runs) is metered honestly: held, settled to actual usage, and refunded when unused