# Web Scraping Data API (https://www.socialcrawl.dev/platforms/web)

> Web Scraping API through SocialCrawl: scrape any public page to clean markdown for 1 credit, search the web, map whole sites, extract structured fields, run async crawls, and schedule change monitors. Every response lands in the unified WebPage schema, and the same x-api-key covers 47 other platforms.

TL;DR: SocialCrawl's Web Scraping API exposes 13 endpoints, 0 to 25 credits per call, all returning one unified JSON schema with AI-enriched fields. Single x-api-key auth, 100 free credits on signup.

## Endpoints

| Endpoint | Method | Path | Credits | Description |
| --- | --- | --- | --- | --- |
| Scrape | GET | `/v1/web/scrape` | 1 | Scrape a web page |
| Search | GET | `/v1/web/search` | 2 | Search the web |
| Map | GET | `/v1/web/map` | 1 | Map URLs on a site |
| Extract | GET | `/v1/web/extract` | 5 | Extract structured data from a web page |
| Start Crawl | POST | `/v1/web/crawl` | 1 | Start an async web crawl |
| Start Batch Scrape | POST | `/v1/web/batch-scrape` | 1 | Start an async batch scrape |
| List Jobs | GET | `/v1/web/jobs` | 0 | List async web jobs |
| Run Agent | POST | `/v1/web/agent` | 25 | Start an async web agent job |
| Create Monitor | POST | `/v1/web/monitors` | 0 | Create a web monitor |
| List Monitors | GET | `/v1/web/monitors` | 0 | List web monitors |
| Open Session | POST | `/v1/web/sessions` | 5 | Create an interactive web session |
| List Sessions | GET | `/v1/web/sessions` | 0 | List interactive web sessions |
| Parse Document | POST | `/v1/web/parse` | 1 | Parse an uploaded document |

## Code example

```bash
curl "https://www.socialcrawl.dev/v1/web/scrape?url=https%3A%2F%2Fexample.com" \
  -H "x-api-key: sc_YOUR_API_KEY"
```

Every response shares one JSON envelope: `{ success, platform, endpoint, data, credits_used, credits_remaining }`.

## SocialCrawl vs DIY scraping stack

| Feature | SocialCrawl | DIY scraping stack |
| --- | --- | --- |
| Setup | One GET request returns clean markdown in a stable schema | Headless browsers, proxy pools, and parsers you build and patch |
| JavaScript rendering | Real browser rendering built in, sessions for interactive flows | Maintain your own Playwright or Puppeteer fleet |
| Change monitoring | Scheduled monitors with per-check billing, down to 5-minute cadence | Cron jobs, diffing logic, and alert plumbing you write yourself |
| Async crawls | Hold-and-settle billing with automatic refunds for unused pages | Queue infrastructure and retry logic you operate |
| Pricing | From 1 credit per page, one balance across 48 platforms | Proxy, compute, and maintenance costs that scale with volume |
| Beyond the open web | Same key reaches social, commerce, and research data | Separate scrapers per platform, each with its own breakage |

## FAQ

### What is the SocialCrawl Web Scraping API?

It is a family of /v1/web/* endpoints that fetch and structure the open web: scrape one page to clean markdown or HTML, search the web, map every URL on a site, extract structured fields with a prompt or JSON schema, run async crawls and batch scrapes, schedule change monitors, and drive interactive browser sessions, all behind the same key, envelope, and credit meter as SocialCrawl's social and commerce data.

### What does a scrape response look like?

Content-bearing endpoints return the unified WebPage schema: the page URL, final URL after redirects, title, description, markdown and HTML content, optional screenshot media, structured extraction output, and fetch metadata, all under data.page in the standard SocialCrawl envelope. List endpoints like search and map return a WebPageList with items[].

### How much does web scraping cost?

A basic page scrape is 1 credit. Web search costs 2 credits, site maps 1 credit, structured extraction 5 credits. Variable work is metered honestly: async crawls hold credits per page and settle to actual usage with automatic refunds for unused pages. New accounts get 100 free credits with no card required.

### Can I crawl a whole site asynchronously?

Yes. POST /v1/web/crawl starts an async job and returns a job id you can poll, list, or cancel. Credits are held up front for the page limit you set, then settled to what the crawl actually fetched. Cancelling refunds the unused hold. Batch scraping many known URLs works the same way through /v1/web/batch-scrape.

### Can I watch a page for changes?

Yes. POST /v1/web/monitors creates a scheduled monitor with a cadence as tight as every 5 minutes. Each check is billed only when it runs, and you can list checks, pause, resume, or delete the monitor at any time through the same /v1/web/monitors endpoints.

### Does JavaScript-heavy content work?

Yes. Pages render in a real browser environment before capture, so client-side content is included. For flows that need clicks, logins, or custom navigation you can open an interactive session with POST /v1/web/sessions and execute browser code in it step by step.

### How is this different from running my own scraper?

No proxy pools, headless browser fleet, retry logic, or parsing pipeline to maintain. One GET request returns clean markdown in a stable schema, and the same key also covers SocialCrawl's social, commerce, and research platforms, so web data lands in the same shape as everything else you collect.

### Does SocialCrawl offer a web data API?

Yes. SocialCrawl's Web Scraping API is a full web data API: a single x-api-key scrapes any public page to clean markdown, searches the web, maps a whole site, extracts structured fields with a prompt or JSON schema, runs async crawls, and schedules change monitors. Every response lands in the unified WebPage schema, so open-web data arrives in the same shape as the social and commerce data on the same key, from 1 credit per page.

### What is the best web scraper API?

SocialCrawl is a strong web scraper API when you want clean output without running the plumbing yourself. Pages render in a real browser before capture, so JavaScript-heavy content is included, and there are no proxy pools, headless browser fleets, or retry logic to maintain. One request returns markdown or HTML in a stable schema from 1 credit per page, and interactive sessions handle flows that need clicks or logins.

## Pricing

- Standard endpoints: 1 credit per call
- Advanced endpoints: 5 credits per call
- Premium endpoints: 10 credits per call
- 100 free credits on signup, no credit card required. Cached responses cost 0 credits. Credit packs never expire.
- Full pricing: https://www.socialcrawl.dev/pricing

## Explore with AI

Questions this API answers, phrased for an AI assistant:

- How to scrape a web page to markdown with the SocialCrawl API
- What does the SocialCrawl web scraping API cost per page?
- Can I monitor a website for changes through an API?