Web Scraping APIScrape pages, search the web, map sites, extract fields, crawl, batch scrape, and run agents with one SocialCrawl API key. Firecrawl upstream under the same envelope and credit system as social. Reads and jobs are priced in credits.
22 active endpoints. 10 primary product surfaces.
Primary product cards cover scrape, search, map, extract, crawl, batch-scrape, agent, parse, sessions, and monitors. The registry also exposes jobs and operational GETs (status, cancel, list) at 0 credit. Firecrawl is the upstream. Same SocialCrawl key as social platforms.
/v1/web/scrapeFetch one public URL as clean markdown or HTML with metadata. Optional formats, proxy, wait_for, and only_main_content. Use for a known URL; start crawl when you need many pages.
url, formats, proxy, wait_for, only_main_content
/v1/web/searchRanked web, news, and image results for a query. Title, URL, and snippet per hit. Limit, country, and domain filters supported.
query, limit, country, include_domains, exclude_domains
/v1/web/mapDiscover URLs on a site without fetching each page body. Optional search filter, limit, and sitemap mode. Use before scrape or crawl.
url, limit, search, sitemap
/v1/web/extractPull structured fields from one page with a JSON schema or plain-language prompt. Prefer this when you need price, author, or plan fields rather than full markdown.
url, schema or prompt
/v1/web/crawlStart an async crawl that walks a site and scrapes pages it finds. Returns job_id. Hold is max(1, limit); settlement follows pages scraped. Poll GET /v1/web/jobs/{job_id}.
url, limit, depth, webhook_url
/v1/web/batch-scrapeScrape a list of known URLs in one async job. Hold equals the number of URLs. Returns job_id for status and results. Prefer crawl when the site should be explored for you.
urls
/v1/web/agentStart a browser agent that follows a plain-language instruction on a site. Returns job_id. 25-credit hold up front, then settle to tokens used. Use when clicking and navigation are required.
url, prompt, model
/v1/web/parseUpload a document such as a PDF and receive clean markdown plus page count. Use scrape for normal web pages; use parse when the input is a file.
file (multipart), mime_type, filename, url
/v1/web/sessionsOpen a short-lived browser session on a URL, then execute interactions while it lives. Minimum 5-credit hold with TTL-based settlement. Close when done to settle remaining hold.
url, ttl_seconds
/v1/web/monitorsCreate a scheduled recheck of a page or search. Creating the monitor costs 0 credits. Each later check is billed when it runs. Pause or delete without losing history rules from the API.
url, cadence, mode, webhook_url
Returns one web page's content as clean markdown or HTML, with the resolved URL, status code, fetch metadata, and an optional screenshot.
Use it for a single known URL; for whole sites start a crawl job instead.
url · Public URL to fetch.
$ curl https://www.socialcrawl.dev/v1/web/scrape?url=https%3A%2F%2Fexample.com&formats=markdown \
-H "x-api-key: sc_YOUR_API_KEY"// Edit the params above and hit "Try it" to run a live request against the APIWeb Scraping is a normal SocialCrawl surface under /v1/web. You authenticate with x-api-key, pay in credits (including async holds that settle when jobs finish), and read one JSON envelope. Firecrawl is the upstream. Same key as social.
Send your key in the x-api-key header. No separate Firecrawl account for customers. The same SocialCrawl key works for Reddit, TikTok, Perplexity, and the rest of the catalog.
scrape, search, map, and extract are GET with query params. crawl, batch-scrape, agent, parse, sessions, and monitors use POST bodies (or multipart for parse). Poll async work with GET /v1/web/jobs/{job_id}.
Sync tiers: scrape/map/parse 1, search 2, extract 5. Async crawl holds max(1, limit) and settles to pages scraped. batch-scrape holds URL count. agent holds 25 then settles tokens. sessions hold at least 5 with TTL settle. Monitor create is 0; checks bill later. Job status GETs are 0 credit.
Every response is the same shape: success, data, credits_used, credits_remaining, request_id, cached. Async starts return a job_id. Status and cancel routes update settlement without inventing a second envelope.
Most products discover URLs, scrape what matters, then crawl only when a whole site is required. Async jobs always end with a status poll.
GET /v1/web/scrape?url=…WebPage: markdown/HTML, url, metadata
Known single URL. Fastest path to clean page content.
GET /v1/web/search?query=…WebPageList: ranked titles, URLs, snippets
When you do not know the URL yet.
POST /v1/web/crawl { url, limit }job_id for a site-wide async crawl
Many pages from one origin with limit and depth controls.
GET /v1/web/jobs/{job_id}Progress, settlement, and results when ready
0-credit GET /v1/web/jobs/{job_id} until complete or cancel.
GET /v1/web/scrape
?url=https://example.com
&formats=markdown
Host: www.socialcrawl.dev
x-api-key: sc_your_api_key_here
POST /v1/web/crawl
Content-Type: application/json
{ "url": "https://docs.example.com", "limit": 25 }{
"success": true,
"data": {
"url": "https://example.com",
"markdown": "# Example…",
"metadata": {}
},
"credits_used": 1,
"credits_remaining": 9999,
"request_id": "req_…",
"cached": false
}Web archetypes sit beside social Post and Author types. Parse code against WebPage, WebPageList, and job envelopes.
url, markdown/html content, metadata, optional screenshot or extraction payload (scrape, parse, extract)
items[] with title, url, snippet or mapped path rows (search, map)
structured fields under extraction shaped by schema or prompt
job_id, kind, status, credits hold/settlement, progress, result refs (crawl, batch-scrape, agent)
Same request lifecycle as every other /v1 platform endpoint. Web is not a separate product stack.
Next.js catch-all routes into the Hono social API. We mint a request_id, authenticate the key, then enforce rate limit and concurrency for the key.
Registry lookup finds the web resource. Param and body validation run first. Invalid input returns 400 with no charge. Sync routes debit the tier cost. Async routes place a credit hold before upstream work.
Eligible search and map paths may hit a short public cache and return credits_used = 0. Scrape and extract typically go live. Async work is handed to Firecrawl jobs; you poll jobs/* for progress.
Upstream payloads map to WebPage, WebPageList, or job status objects, validate against schema, wrap the success envelope, and settle or refund holds when jobs finish or fail.
Billing rules that matter in production
Web Scraping is public page and SERP access through Firecrawl, normalized into SocialCrawl WebPage and job envelopes so your code keeps one key and one billing model.
Public web pages, search results, site maps, structured extraction, multi-page crawls, batch URL scrapes, browser agents, document parse, sessions, and change monitors. Not social profile or comment trees.
Firecrawl is the upstream for scrape, search, map, extract, crawl, batch-scrape, agent, and related job operations. Customers do not manage a separate Firecrawl key for the SocialCrawl path.
A unified JSON envelope: success, data, credits_used, request_id, cached. Page content lands as WebPage. Search and map land as WebPageList. Async starts return job_id for status and settlement.
No unlimited free crawl. Holds and settlement always apply to async work. Job status GETs are free but do not replace the cost of pages scraped. Private authenticated sites still need sessions or agent flows where the product supports them.
The jobs this API is most often used for.
Page scrape, site crawl, and change monitoring
Callers concentrate on single-URL scrape, web search into scrape chains, and crawl or batch-scrape when inventory is large. Monitors appear for recurring change checks. Agent and sessions show up when navigation or login steps block a simple GET. Firecrawl-powered web under the same SocialCrawl key and envelope as social. Clear async hold and settle rules, with jobs/* status at 0 credit.
Single scrape is typically seconds on a live miss. Search and extract can be slower. Crawl, batch-scrape, and agent are async: poll jobs until settlement.
The jobs Web Scraping data is most often used for. Each one is a full recipe with the endpoint chain and pricing.
Common ways teams put this data to work, and the stack each one tends to run.
Scrape known URLs, map then crawl docs sites, and land markdown into warehouses. Poll jobs for multi-page runs.
search for candidates, scrape winners, extract pricing or author fields. Pair with Perplexity or Tavily when a synthesised answer is enough first.
Monitors for change detection, sessions for interactive pages, agent when multi-step browser work is required. Same key as social monitoring.
Start with scrape at 1 credit or search at 2. For multi-page work, POST crawl or batch-scrape and poll jobs at 0 credit until settlement.
curl "https://www.socialcrawl.dev/v1/web/scrape?url=https://example.com&formats=markdown" \
-H "x-api-key: sc_your_api_key_here"curl "https://www.socialcrawl.dev/v1/web/search?query=social+media+api&limit=10" \
-H "x-api-key: sc_your_api_key_here"Same key as the rest of the catalog
Every endpoint returns structured JSON in a unified envelope. Computed fields such as engagement rate and content category are included only where the endpoint supports them and the required source inputs are present.
Same Web Scraping data, two very different paths. Auth, rate limits, schema, and cost — side by side.
| Feature | SocialCrawl | DIY scraping stack |
|---|---|---|
| Setup | One GET request returns clean markdown in a stable schema | Headless browsers, proxy pools, and parsers you build and patch |
| JavaScript rendering | Real browser rendering built in, sessions for interactive flows | Maintain your own Playwright or Puppeteer fleet |
| Change monitoring | Scheduled monitors with per-check billing, down to 5-minute cadence | Cron jobs, diffing logic, and alert plumbing you write yourself |
| Async crawls | Hold-and-settle billing with automatic refunds for unused pages | Queue infrastructure and retry logic you operate |
| Pricing | From 1 credit per page, one balance across 50 platforms | Proxy, compute, and maintenance costs that scale with volume |
| Beyond the open web | Same key reaches social, commerce, and research data | Separate scrapers per platform, each with its own breakage |
Setup
JavaScript rendering
Change monitoring
Async crawls
Pricing
Beyond the open web
Find answers to frequently asked questions about SocialCrawl's API, pricing, and capabilities.
Contact usAsk AI about SocialCrawl