100 free credits — no credit card required

Web Crawl API

Name: Web Crawl API
Brand: SocialCrawl
Availability: InStock

Scrape Web Crawl data with one API call. Starts an async crawl job and returns a SocialCrawl job id for polling, cancellation, and webhook tracking.

Last updated July 2026

Start for free View Documentation

Try the Web Crawl API

See real data before writing a single line

GET/v1/web/crawl

Searching 44 platforms in parallel

TikTok·

Instagram·

YouTube·

Facebook·

X·

LinkedIn·

Reddit·

Threads·

Pinterest·

Twitch·

Truth Social·

Snapchat·

Kick·

Bluesky·

Kwai·

Rumble·

Spotify·

TikTok Shop·

Amazon Shop·

Google Shopping·

Trustpilot·

TripAdvisor·

Linktree·

Komi·

Pillar·

lnk.bio·

Facebook Ads·

Google Ads·

LinkedIn Ads·

Google Search·

Google News·

Google Finance·

Polymarket·

Tavily·

Hacker News·

GitHub·

Perplexity·

Naver·Utility·

Universal Search

TikTok·

Instagram·

YouTube·

Facebook·

X·

LinkedIn·

Reddit·

Threads·

Pinterest·

Twitch·

Truth Social·

Snapchat·

Kick·

Bluesky·

Kwai·

Rumble·

Spotify·

TikTok Shop·

Amazon Shop·

Google Shopping·

Trustpilot·

TripAdvisor·

Linktree·

Komi·

Pillar·

lnk.bio·

Facebook Ads·

Google Ads·

LinkedIn Ads·

Google Search·

Google News·

Google Finance·

Polymarket·

Tavily·

Hacker News·

GitHub·

Perplexity·

Naver·Utility·

Universal Search

TikTok·

Instagram·

YouTube·

Facebook·

X·

LinkedIn·

Reddit·

Threads·

Pinterest·

Twitch·

Truth Social·

Snapchat·

Kick·

Bluesky·

Kwai·

Rumble·

Spotify·

TikTok Shop·

Amazon Shop·

Google Shopping·

Trustpilot·

TripAdvisor·

Linktree·

Komi·

Pillar·

lnk.bio·

Facebook Ads·

Google Ads·

LinkedIn Ads·

Google Search·

Google News·

Google Finance·

Polymarket·

Tavily·

Hacker News·

GitHub·

Perplexity·

Naver·Utility·

Universal Search

TikTok·

Instagram·

YouTube·

Facebook·

X·

LinkedIn·

Reddit·

Threads·

Pinterest·

Twitch·

Truth Social·

Snapchat·

Kick·

Bluesky·

Kwai·

Rumble·

Spotify·

TikTok Shop·

Amazon Shop·

Google Shopping·

Trustpilot·

TripAdvisor·

Linktree·

Komi·

Pillar·

lnk.bio·

Facebook Ads·

Google Ads·

LinkedIn Ads·

Google Search·

Google News·

Google Finance·

Polymarket·

Tavily·

Hacker News·

GitHub·

Perplexity·

Naver·Utility·

Universal Search

Web API

What can you do with the Crawl API?

The Crawl endpoint gives you structured Web data with computed fields in a single request. No scraping infrastructure to build or maintain.

Example Request

curl -H "x-api-key: YOUR_API_KEY" \
  "https://www.socialcrawl.dev/v1/web/crawl?url=https%3A%2F%2Fexample.com"

import requests

response = requests.get(
    "https://www.socialcrawl.dev/v1/web/crawl",
    params={
    'url': 'https://example.com',
    },
    headers={"x-api-key": "YOUR_API_KEY"},
)

data = response.json()

const response = await fetch(
  "https://www.socialcrawl.dev/v1/web/crawl?url=https%3A%2F%2Fexample.com",
  {
    headers: { "x-api-key": "YOUR_API_KEY" },
  },
);

const data = await response.json();

Parameters

Parameter	Required	Description
url	Yes	Root URL to crawl.
limit	No	Maximum pages to crawl.
max_depth	No
allow_backward_links	No
allow_external_links	No
include_paths	No
exclude_paths	No
webhook_url	No	Optional webhook URL for terminal job updates.
formats	No

Example Response

What does the Web Crawl API return?

Every response follows one unified schema. Here is a real, unmodified response body, so you can see the exact fields you get back before spending a credit.

Example response

{
  "success": true,
  "platform": "instagram",
  "endpoint": "/v1/instagram/engagement",
  "data": {
    "engagement_rate_percentages": 38.33,
    "recent_posts": 12,
    "followers": 87608035,
    "comments": 528912,
    "likes": 33049046,
    "recent_posts_explanation": "Statistics based on the last 12 posts",
    "id_user": "2278169415",
    "username": "mrbeast",
    "is_private": false,
    "posts_details": [
      {
        "likes": 5636982,
        "comments": 69484,
        "taken_at": 1781457954,
        "datetime": "2026-06-14 20:25:54",
        "hours_since_post": 461,
        "time_ago": "19 days ago",
        "likes_per_hour": 12228,
        "comments_per_hour": 151
      },
      {
        "likes": 20000768,
        "comments": 223510,
        "taken_at": 1732824650,
        "datetime": "2024-11-28 23:10:50",
        "hours_since_post": 13971,
        "time_ago": "2 years ago",
        "likes_per_hour": 1432,
        "comments_per_hour": 16
      },
      {
        "likes": 929226,
        "comments": 30633,
        "taken_at": 1782232475,
        "datetime": "2026-06-23 19:34:35",
        "hours_since_post": 246,
        "time_ago": "10 days ago",
        "likes_per_hour": 3777,
        "comments_per_hour": 125
      },
      {
        "likes": 487761,
        "comments": 22482,
        "taken_at": 1781799425,
        "datetime": "2026-06-18 19:17:05",
        "hours_since_post": 366,
        "time_ago": "15 days ago",
        "likes_per_hour": 1333,
        "comments_per_hour": 61
      },
      {
        "likes": 712265,
        "comments": 15716,
        "taken_at": 1781366405,
        "datetime": "2026-06-13 19:00:05",
        "hours_since_post": 487,
        "time_ago": "20 days ago",
        "likes_per_hour": 1463,
        "comments_per_hour": 32
      },
      {
        "likes": 1475116,
        "comments": 35386,
        "taken_at": 1781277094,
        "datetime": "2026-06-12 18:11:34",
        "hours_since_post": 512,
        "time_ago": "21 days ago",
        "likes_per_hour": 2881,
        "comments_per_hour": 69
      },
      {
        "likes": 1108220,
        "comments": 26632,
        "taken_at": 1780160249,
        "datetime": "2026-05-30 19:57:29",
        "hours_since_post": 822,
        "time_ago": "1 months ago",
        "likes_per_hour": 1348,
        "comments_per_hour": 32
      },
      {
        "likes": 542948,
        "comments": 28476,
        "taken_at": 1779375582,
        "datetime": "2026-05-21 17:59:42",
        "hours_since_post": 1040,
        "time_ago": "1 months ago",
        "likes_per_hour": 522,
        "comments_per_hour": 27
      },
      {
        "likes": 698514,
        "comments": 24401,
        "taken_at": 1779120014,
        "datetime": "2026-05-18 19:00:14",
        "hours_since_post": 1111,
        "time_ago": "2 months ago",
        "likes_per_hour": 629,
        "comments_per_hour": 22
      },
      {
        "likes": 468000,
        "comments": 13548,
        "taken_at": 1778947209,
        "datetime": "2026-05-16 19:00:09",
        "hours_since_post": 1159,
        "time_ago": "2 months ago",
        "likes_per_hour": 404,
        "comments_per_hour": 12
      },
      {
        "likes": 526594,
        "comments": 24411,
        "taken_at": 1777737719,
        "datetime": "2026-05-02 19:01:59",
        "hours_since_post": 1495,
        "time_ago": "2 months ago",
        "likes_per_hour": 352,
        "comments_per_hour": 16
      },
      {
        "likes": 462652,
        "comments": 14233,
        "taken_at": 1777580305,
        "datetime": "2026-04-30 23:18:25",
        "hours_since_post": 1538,
        "time_ago": "2 months ago",
        "likes_per_hour": 301,
        "comments_per_hour": 9
      }
    ]
  },
  "credits_used": 5,
  "credits_remaining": 9999,
  "request_id": "req_example000000",
  "cached": false
}

Live sample illustrating the unified response shape. Field values reflect the record you query.

API Details

How does the Web Crawl API work?

Send a GET request with your API key and get back clean, structured JSON. Every response follows our unified schema with computed fields.

Method

GET

Response

JSON

Why SocialCrawl

Why use SocialCrawl for Web Crawl data?

We handle the complexity of Web data extraction so you can focus on building. Unified schema, AI enrichment, and zero platform logic in your code.

One schema for every platform — write your integration once
Computed fields on every response: engagement rate, reach, category, language
No scraping infrastructure to build, no proxies to manage, no rate limits to fight
Real-time data from Web — no caching, no delays

Developer First

How do you scrape social media data in seconds?

The fastest social media scraping API for developers. Scrape profiles, posts, comments, and analytics from 44 platforms covering 10B+ monthly active users.

One schema, every platform

Query 44 platforms with identical response structures. Write your integration once.

Computed fields, not just scraped

Every response includes engagement_rate, estimated_reach, content_category, and language — ready to use.

See your data before you code

Visual Data Explorer — paste any URL, get rich result cards, sortable tables, CSV export.

import requests

response = requests.get(
    'https://www.socialcrawl.dev/v1/tiktok/profile',
    params={'handle': 'charlidamelio'},
    headers={'x-api-key': 'sc_YOUR_API_KEY'}
)
data = response.json()

[ .JSON ]

{
  "success": true,
  "platform": "tiktok",
  "data": {
    "author": {
      "username": "charlidamelio",
      "followers": 152400000
    },
    "engagement": {
      "likes": 12400000000,
      "engagement_rate": 0.087
    },
    "metadata": {
      "language": "en",
      "content_category": "lifestyle"
    }
  }
}

+ 44 platforms

Ready to scrape Web Crawl data?

Get your API key and start pulling Web data in under 60 seconds.

Start for free