Web Scraping Scrape API
Scrape Web Scraping Scrape data with one API call. Fetches a public web page and returns clean content, metadata, and optional media in the unified WebPage schema.
Last updated October 2026Maintained by the SocialCrawl team
Returns one web page's content as clean markdown or HTML, with the resolved URL, status code, fetch metadata, and an optional screenshot.
Use it for a single known URL; for whole sites start a crawl job instead.
Try the Web Scraping Scrape API
See real data before writing a single line
Searching 68 platforms in parallel
TikTok·
Instagram·
YouTube·
Facebook·
X·
LinkedIn·
Reddit·
Threads·
Pinterest·
Twitch·
Truth Social·
Snapchat·
Kick·
Bluesky·
Kwai·
Rumble·
Spotify·
Apple Music·TikTok Shop·
Amazon Shop·
Google Shopping·
Trustpilot·
TripAdvisor·
Yelp·
Linktree·
Komi·
Pillar·
lnk.bio·Facebook Ads·Google Ads·LinkedIn Ads·Google Search·Google News·
Finance·
Polymarket·
Tavily·
Hacker News·
GitHub·
Perplexity·
Naver·
Utility·
Universal SearchTikTok·Instagram·YouTube·Facebook·X·LinkedIn·Reddit·Threads·Pinterest·Twitch·Truth Social·Snapchat·Kick·Bluesky·Kwai·Rumble·Spotify·Apple Music·TikTok Shop·Amazon Shop·Google Shopping·Trustpilot·TripAdvisor·Yelp·Linktree·Komi·Pillar·lnk.bio·Facebook Ads·Google Ads·LinkedIn Ads·Google Search·Google News·Finance·Polymarket·Tavily·Hacker News·GitHub·Perplexity·Naver·Utility·Universal SearchTikTok·Instagram·YouTube·Facebook·X·LinkedIn·Reddit·Threads·Pinterest·Twitch·Truth Social·Snapchat·Kick·Bluesky·Kwai·Rumble·Spotify·Apple Music·TikTok Shop·Amazon Shop·Google Shopping·Trustpilot·TripAdvisor·Yelp·Linktree·Komi·Pillar·lnk.bio·Facebook Ads·Google Ads·LinkedIn Ads·Google Search·Google News·Finance·Polymarket·Tavily·Hacker News·GitHub·Perplexity·Naver·Utility·Universal SearchTikTok·Instagram·YouTube·Facebook·X·LinkedIn·Reddit·Threads·Pinterest·Twitch·Truth Social·Snapchat·Kick·Bluesky·Kwai·Rumble·Spotify·Apple Music·TikTok Shop·Amazon Shop·Google Shopping·Trustpilot·TripAdvisor·Yelp·Linktree·Komi·Pillar·lnk.bio·Facebook Ads·Google Ads·LinkedIn Ads·Google Search·Google News·Finance·Polymarket·Tavily·Hacker News·GitHub·Perplexity·Naver·Utility·Universal SearchWhat can you do with the Scrape API?
The Scrape endpoint gives you structured Web Scraping data with computed fields in a single request. No scraping infrastructure to build or maintain.
Example Request
curl -H "x-api-key: YOUR_API_KEY" \
"https://www.socialcrawl.dev/v1/web/scrape?url=https%3A%2F%2Fexample.com&formats=markdown"import requests
response = requests.get(
"https://www.socialcrawl.dev/v1/web/scrape",
params={
'url': 'https://example.com',
'formats': 'markdown',
},
headers={"x-api-key": "YOUR_API_KEY"},
)
data = response.json()const response = await fetch(
"https://www.socialcrawl.dev/v1/web/scrape?url=https%3A%2F%2Fexample.com&formats=markdown",
{
headers: { "x-api-key": "YOUR_API_KEY" },
},
);
const data = await response.json();Parameters
| Parameter | Required | Description |
|---|---|---|
| url | Yes | Public URL to fetch. |
| formats | No | Comma-separated output formats such as markdown,screenshot. |
| only_main_content | No | Strip nav, headers, footers and sidebars and keep the article body. Default true; send false when you need the whole page, including the chrome. A markdown-only scrape of a site the primary scraper cannot fetch is served by a second extractor, which returns the whole page. |
| wait_for | No | Milliseconds to wait after load before capturing, for pages that render their content in JavaScript. Leave unset unless the page comes back empty or half-built. |
| mobile | No | Render in a mobile viewport with a mobile user agent. Use it when the site serves a different layout to phones. |
| timeout | No | Hard cap on the page load in milliseconds, 1 to 30000. A page that exceeds it fails and is refunded rather than returning a partial capture. |
| max_age | No | Accept an upstream-cached copy up to this many milliseconds old, which is faster and still one credit. Default 172800000 (48 hours); send 0 to force a live fetch. |
| location_country | No | ISO 3166-1 alpha-2 country code to fetch from, e.g. 'us' or 'de'. Use it for geo-varying pages such as pricing or availability. |
| screenshot_full_page | No | When a screenshot format is requested, capture the entire scrollable page instead of the visible viewport. |
| include_tags | No | CSV of CSS selectors or HTML tags to keep, e.g. 'article,main'. Everything outside them is dropped. Use it to pin extraction to a known container. |
| exclude_tags | No | CSV of CSS selectors or HTML tags to drop, e.g. 'nav,footer,.cookie-banner'. Applied after include_tags. |
| proxy | No | Proxy tier: basic, auto, or enhanced. (basic | auto | enhanced) |
| pdf_parse | No | Parse the target as a PDF and return its extracted text. Send it when the URL points at a PDF rather than an HTML page. |
| block_ads | No | Block ad and tracker requests, which is faster and quieter. Default true. |
| remove_base64_images | No | Drop inline base64-encoded images from the returned content instead of carrying them in the payload. Default true. |
What does the Web Scraping Scrape API return?
Every response follows one unified schema. Here is a real, unmodified response body, so you can see the exact fields you get back before spending a credit.
Example response
{
"success": true,
"platform": "web",
"endpoint": "/v1/web/scrape",
"data": {
"page": {
"url": "https://example.com",
"final_url": "https://example.com/",
"status_code": 200,
"scrape_id": "01a0fca4-f04b-7369-9cd6-64e0f4d82214",
"fetched_at": null,
"content": {
"markdown": "Thisdomainisforuseindocumentationexampleswithoutneedingpermission.Thisisnotaservice,avoidrelyingonitfortestingandmonitoringpurposes.\n\nهذا النطاق مُخصص للاستخدام في أمثلة التوثيق دون الحاجة إلى إذن. هذه ليست خدمة، يُرجى تجنب الاعتماد عليها لأغراض الاختبار والمراقبة.\n\n该域名仅用于文档示例，无需获得许可。这并非一项服务，请勿将其用于测试和监控目的。\n\nL’usagedecedomaineestréservéàdesexemplesdedocumentation,sansautorisationpréalable.Ilnes’agitpasd’unservice;sonutilisationàdesfinsdetestoudesurveillanceestàéviter.\n\nДанныйдоменпредназначендляиспользованиявпримерахдокументациибезнеобходимостиполученияпредварительногоразрешения.Этонесервис;нерекомендуетсяегоиспользованиедлятестированияимониторинга.\n\nEstedominioestádestinadoalusoenejemplosdedocumentaciónsinnecesidaddepermiso.Estonoesunservicio,evitarutilizarlopararealizarpruebasomonitoreos.\n\n [Learn more](https://iana.org/help/example-domains)",
"html": null,
"raw_html": null,
"summary": null
},
"media": {
"screenshot_url": null,
"audio_url": null,
"video_url": null
},
"extraction": null,
"answer": null,
"highlights": null,
"change_tracking": null,
"page_count": null,
"total_page_count": null,
"fetch": {
"cache_state": "hit",
"cached_at": "2026-10-02T12:40:55.009Z",
"proxy_tier": null
},
"page_state": {
"value": "content",
"confidence": 0.85,
"skipped_judge": false
}
}
},
"credits_used": 1,
"request_id": "req_example000000",
"cached": false,
"credits_remaining": 9999,
"source": "captured",
"captured_at": "2026-10-02T12:44:23.265Z",
"redacted": true
}Live sample illustrating the unified response shape. Field values reflect the record you query.
How does the Web Scraping Scrape API work?
Send a GET request with your API key and get back clean, structured JSON in our unified schema. Supported computed fields are populated when the source provides the required inputs.
Method
GET
Response
JSON
How do you scrape social media data in seconds?
The fastest social media scraping API for developers. Scrape profiles, posts, comments, and analytics from 68 platforms covering 10B+ monthly active users.
One schema, every platform
Query 68 platforms with identical response structures. Write your integration once.
Computed fields, not just scraped
When an endpoint supports these metrics and the source provides the required inputs, the normalized record includes engagement_rate, estimated_reach, content_category, and language. Ready to use.
See your data before you code
Visual Data Explorer. Paste any URL, get rich result cards, sortable tables, CSV export.
import requests
response = requests.get(
'https://www.socialcrawl.dev/v1/tiktok/profile',
params={'handle': 'charlidamelio'},
headers={'x-api-key': 'sc_YOUR_API_KEY'}
)
data = response.json(){
"success": true,
"platform": "tiktok",
"data": {
"author": {
"username": "charlidamelio",
"followers": 152400000
},
"engagement": {
"likes": 12400000000,
"engagement_rate": 0.087
},
"metadata": {
"language": "en",
"content_category": "lifestyle"
}
}
}Ready to scrape Web Scraping Scrape data?
Get your API key and start pulling Web Scraping data in under 60 seconds.