# Web Data API (https://www.socialcrawl.dev/platforms/web)

> Web Scraping API through SocialCrawl: scrape any public page to clean markdown for 1 credit, search the web, map whole sites, extract structured fields, run async crawls, and schedule change monitors. Every response lands in the unified WebPage schema, and the same x-api-key covers 43 other platforms.

TL;DR: SocialCrawl's Web API exposes 13 endpoints, 0 to 25 credits per call, all returning one unified JSON schema with AI-enriched fields. Single x-api-key auth, 100 free credits on signup.

## Endpoints

| Endpoint | Method | Path | Credits | Description |
| --- | --- | --- | --- | --- |
| Scrape | GET | `/v1/web/scrape` | 1 | Scrape a web page |
| Search | GET | `/v1/web/search` | 2 | Search the web |
| Map | GET | `/v1/web/map` | 1 | Map URLs on a site |
| Extract | GET | `/v1/web/extract` | 5 | Extract structured data from a web page |
| Start Crawl | POST | `/v1/web/crawl` | 1 | Start an async web crawl |
| Start Batch Scrape | POST | `/v1/web/batch-scrape` | 1 | Start an async batch scrape |
| List Jobs | GET | `/v1/web/jobs` | 0 | List async web jobs |
| Run Agent | POST | `/v1/web/agent` | 25 | Start an async web agent job |
| Create Monitor | POST | `/v1/web/monitors` | 0 | Create a web monitor |
| List Monitors | GET | `/v1/web/monitors` | 0 | List web monitors |
| Open Session | POST | `/v1/web/sessions` | 5 | Create an interactive web session |
| List Sessions | GET | `/v1/web/sessions` | 0 | List interactive web sessions |
| Parse Document | POST | `/v1/web/parse` | 1 | Parse an uploaded document |

## Code example

```bash
curl "https://www.socialcrawl.dev/v1/web/scrape?url=https%3A%2F%2Fexample.com" \
  -H "x-api-key: sc_YOUR_API_KEY"
```

Every response shares one JSON envelope: `{ success, platform, endpoint, data, credits_used, credits_remaining }`.

## SocialCrawl vs DIY scraping stack

| Feature | SocialCrawl | DIY scraping stack |
| --- | --- | --- |
| Setup | One GET request returns clean markdown in a stable schema | Headless browsers, proxy pools, and parsers you build and patch |
| JavaScript rendering | Real browser rendering built in, sessions for interactive flows | Maintain your own Playwright or Puppeteer fleet |
| Change monitoring | Scheduled monitors with per-check billing, down to 5-minute cadence | Cron jobs, diffing logic, and alert plumbing you write yourself |
| Async crawls | Hold-and-settle billing with automatic refunds for unused pages | Queue infrastructure and retry logic you operate |
| Pricing | From 1 credit per page, one balance across 44 platforms | Proxy, compute, and maintenance costs that scale with volume |
| Beyond the open web | Same key reaches social, commerce, and research data | Separate scrapers per platform, each with its own breakage |

## FAQ

### What is the SocialCrawl Web Scraping API?

It is a family of /v1/web/* endpoints that fetch and structure the open web: scrape one page to clean markdown or HTML, search the web, map every URL on a site, extract structured fields with a prompt or JSON schema, run async crawls and batch scrapes, schedule change monitors, and drive interactive browser sessions — all behind the same key, envelope, and credit meter as SocialCrawl's social and commerce data.

### What does a scrape response look like?

Content-bearing endpoints return the unified WebPage schema: the page URL, final URL after redirects, title, description, markdown and HTML content, optional screenshot media, structured extraction output, and fetch metadata — all under data.page in the standard SocialCrawl envelope. List endpoints like search and map return a WebPageList with items[].

### How much does web scraping cost?

A basic page scrape is 1 credit. Web search costs 2 credits, site maps 1 credit, structured extraction 5 credits. Variable work is metered honestly: async crawls hold credits per page and settle to actual usage with automatic refunds for unused pages. New accounts get 100 free credits with no card required.

### Can I crawl a whole site asynchronously?

Yes. POST /v1/web/crawl starts an async job and returns a job id you can poll, list, or cancel. Credits are held up front for the page limit you set, then settled to what the crawl actually fetched — cancelling refunds the unused hold. Batch scraping many known URLs works the same way through /v1/web/batch-scrape.

### Can I watch a page for changes?

Yes. POST /v1/web/monitors creates a scheduled monitor with a cadence as tight as every 5 minutes. Each check is billed only when it runs, and you can list checks, pause, resume, or delete the monitor at any time through the same /v1/web/monitors endpoints.

### Does JavaScript-heavy content work?

Yes — pages render in a real browser environment before capture, so client-side content is included. For flows that need clicks, logins, or custom navigation you can open an interactive session with POST /v1/web/sessions and execute browser code in it step by step.

### How is this different from running my own scraper?

No proxy pools, headless browser fleet, retry logic, or parsing pipeline to maintain. One GET request returns clean markdown in a stable schema, and the same key also covers SocialCrawl's social, commerce, and research platforms — so web data lands in the same shape as everything else you collect.

### Does SocialCrawl offer a web data API?

Yes. SocialCrawl's Web Scraping API is a full web data API: a single x-api-key scrapes any public page to clean markdown, searches the web, maps a whole site, extracts structured fields with a prompt or JSON schema, runs async crawls, and schedules change monitors. Every response lands in the unified WebPage schema, so open-web data arrives in the same shape as the social and commerce data on the same key, from 1 credit per page.

### What is the best web scraper API?

SocialCrawl is a strong web scraper API when you want clean output without running the plumbing yourself. Pages render in a real browser before capture, so JavaScript-heavy content is included, and there are no proxy pools, headless browser fleets, or retry logic to maintain. One request returns markdown or HTML in a stable schema from 1 credit per page, and interactive sessions handle flows that need clicks or logins.

## Pricing

- Standard endpoints: 1 credit per call
- Advanced endpoints: 5 credits per call
- Premium endpoints: 10 credits per call
- 100 free credits on signup, no credit card required. Cached responses cost 0 credits. Credit packs never expire.
- Full pricing: https://www.socialcrawl.dev/pricing

## Explore with AI

Questions this API answers, phrased for an AI assistant:

- How to scrape a web page to markdown with the SocialCrawl API
- What does the SocialCrawl web scraping API cost per page?
- Can I monitor a website for changes through an API?