Web Scraping Map API
Scrape Web Scraping Map data with one API call. Discovers URLs for a public site and returns them as a normalized WebPageList.
Last updated October 2026Maintained by the SocialCrawl team
Returns a list of URLs that exist on a site, without fetching the content of each page.
Use it to discover what pages a site has before choosing which ones to scrape or crawl.
Try the Web Scraping Map API
See real data before writing a single line
Searching 68 platforms in parallel
TikTok·
Instagram·
YouTube·
Facebook·
X·
LinkedIn·
Reddit·
Threads·
Pinterest·
Twitch·
Truth Social·
Snapchat·
Kick·
Bluesky·
Kwai·
Rumble·
Spotify·
Apple Music·TikTok Shop·
Amazon Shop·
Google Shopping·
Trustpilot·
TripAdvisor·
Yelp·
Linktree·
Komi·
Pillar·
lnk.bio·Facebook Ads·Google Ads·LinkedIn Ads·Google Search·Google News·
Finance·
Polymarket·
Tavily·
Hacker News·
GitHub·
Perplexity·
Naver·
Utility·
Universal SearchTikTok·Instagram·YouTube·Facebook·X·LinkedIn·Reddit·Threads·Pinterest·Twitch·Truth Social·Snapchat·Kick·Bluesky·Kwai·Rumble·Spotify·Apple Music·TikTok Shop·Amazon Shop·Google Shopping·Trustpilot·TripAdvisor·Yelp·Linktree·Komi·Pillar·lnk.bio·Facebook Ads·Google Ads·LinkedIn Ads·Google Search·Google News·Finance·Polymarket·Tavily·Hacker News·GitHub·Perplexity·Naver·Utility·Universal SearchTikTok·Instagram·YouTube·Facebook·X·LinkedIn·Reddit·Threads·Pinterest·Twitch·Truth Social·Snapchat·Kick·Bluesky·Kwai·Rumble·Spotify·Apple Music·TikTok Shop·Amazon Shop·Google Shopping·Trustpilot·TripAdvisor·Yelp·Linktree·Komi·Pillar·lnk.bio·Facebook Ads·Google Ads·LinkedIn Ads·Google Search·Google News·Finance·Polymarket·Tavily·Hacker News·GitHub·Perplexity·Naver·Utility·Universal SearchTikTok·Instagram·YouTube·Facebook·X·LinkedIn·Reddit·Threads·Pinterest·Twitch·Truth Social·Snapchat·Kick·Bluesky·Kwai·Rumble·Spotify·Apple Music·TikTok Shop·Amazon Shop·Google Shopping·Trustpilot·TripAdvisor·Yelp·Linktree·Komi·Pillar·lnk.bio·Facebook Ads·Google Ads·LinkedIn Ads·Google Search·Google News·Finance·Polymarket·Tavily·Hacker News·GitHub·Perplexity·Naver·Utility·Universal SearchWhat can you do with the Map API?
The Map endpoint gives you structured Web Scraping data with computed fields in a single request. No scraping infrastructure to build or maintain.
Example Request
curl -H "x-api-key: YOUR_API_KEY" \
"https://www.socialcrawl.dev/v1/web/map?url=https%3A%2F%2Fexample.com&limit=100"import requests
response = requests.get(
"https://www.socialcrawl.dev/v1/web/map",
params={
'url': 'https://example.com',
'limit': '100',
},
headers={"x-api-key": "YOUR_API_KEY"},
)
data = response.json()const response = await fetch(
"https://www.socialcrawl.dev/v1/web/map?url=https%3A%2F%2Fexample.com&limit=100",
{
headers: { "x-api-key": "YOUR_API_KEY" },
},
);
const data = await response.json();Parameters
| Parameter | Required | Description |
|---|---|---|
| url | Yes | Site URL to map. |
| search | No | Optional path or keyword filter. |
| limit | No | Maximum URLs to return, up to 5000. |
| sitemap | No | How to use the site's sitemap.xml: 'include' (default) to merge it with crawled links, 'only' to trust it alone, 'skip' to ignore it. (include | skip | only) |
| include_subdomains | No | Follow links onto subdomains of the root URL. Default true. |
| ignore_query_parameters | No | Treat URLs differing only by query string as one URL, which keeps paginated and tracking-tagged duplicates out of the result. Default true. |
| fresh | No | Bypass the upstream cache and re-discover the site now. Slower; use it when the map is known to be stale. |
| goal | No | Optional. What you are looking for on the site, in your own words. The map still returns every URL, ranked, with skipped_urls listing the ones that look unlikely to hold it (each with p). Without this param the map is unchanged. |
What does the Web Scraping Map API return?
Every response follows one unified schema. Here is a real, unmodified response body, so you can see the exact fields you get back before spending a credit.
Example response
{
"success": true,
"platform": "web",
"endpoint": "/v1/web/map",
"data": {
"items": [
{
"page": {
"url": "https://example.com/",
"final_url": "https://example.com/",
"status_code": null,
"scrape_id": null,
"fetched_at": null,
"title": "Example Domain",
"description": null,
"content": {
"markdown": null,
"html": null,
"raw_html": null,
"summary": null
},
"media": {
"screenshot_url": null,
"audio_url": null,
"video_url": null
},
"extraction": null,
"answer": null,
"highlights": null,
"change_tracking": null,
"page_count": null,
"total_page_count": null,
"fetch": {
"cache_state": null,
"cached_at": null,
"proxy_tier": null
}
}
},
{
"page": {
"url": "https://www.example.com/",
"final_url": "https://www.example.com/",
"status_code": null,
"scrape_id": null,
"fetched_at": null,
"title": "Example Domain",
"description": null,
"content": {
"markdown": null,
"html": null,
"raw_html": null,
"summary": null
},
"media": {
"screenshot_url": null,
"audio_url": null,
"video_url": null
},
"extraction": null,
"answer": null,
"highlights": null,
"change_tracking": null,
"page_count": null,
"total_page_count": null,
"fetch": {
"cache_state": null,
"cached_at": null,
"proxy_tier": null
}
}
}
],
"total": 3,
"dropped": 0
},
"credits_used": 1,
"request_id": "req_example000000",
"cached": false,
"pagination": {
"next_cursor": null,
"has_more": false,
"page_size": 3
},
"credits_remaining": 9999,
"source": "captured",
"captured_at": "2026-10-02T12:44:25.365Z",
"redacted": true
}Live sample illustrating the unified response shape. Field values reflect the record you query.
How does the Web Scraping Map API work?
Send a GET request with your API key and get back clean, structured JSON in our unified schema. Supported computed fields are populated when the source provides the required inputs.
Method
GET
Response
JSON
How do you scrape social media data in seconds?
The fastest social media scraping API for developers. Scrape profiles, posts, comments, and analytics from 68 platforms covering 10B+ monthly active users.
One schema, every platform
Query 68 platforms with identical response structures. Write your integration once.
Computed fields, not just scraped
When an endpoint supports these metrics and the source provides the required inputs, the normalized record includes engagement_rate, estimated_reach, content_category, and language. Ready to use.
See your data before you code
Visual Data Explorer. Paste any URL, get rich result cards, sortable tables, CSV export.
import requests
response = requests.get(
'https://www.socialcrawl.dev/v1/tiktok/profile',
params={'handle': 'charlidamelio'},
headers={'x-api-key': 'sc_YOUR_API_KEY'}
)
data = response.json(){
"success": true,
"platform": "tiktok",
"data": {
"author": {
"username": "charlidamelio",
"followers": 152400000
},
"engagement": {
"likes": 12400000000,
"engagement_rate": 0.087
},
"metadata": {
"language": "en",
"content_category": "lifestyle"
}
}
}Ready to scrape Web Scraping Map data?
Get your API key and start pulling Web Scraping data in under 60 seconds.