Web Scraping Search API
Returns ranked web, news, and image results for a search query as one normalized list, with a title, URL, and snippet for each.
GET /v1/web/searchLast updated November 2026Maintained by the SocialCrawl team
Try the Web Scraping Search API
See real data before writing a single line
When to use the Web Scraping Search API
Use it when you do not know yet which URLs you need; once you have a URL, scrape it.
From the API reference
Searches web, news, and image sources and returns a single normalized list of web page results. When the primary index cannot answer, a second web index serves a web or news search in the same shape and at the same price: url, title, description, source_type and the news publish time are filled, rows are ranked by that index, and with sort_by_date the news rows are ordered newest first. It does not serve images, categories, location, include_content, more than 20 results a source or a time_range other than qdr:d, qdr:w, qdr:m and qdr:y.
95 platforms behind one API
What can you do with the Search API?
The Search endpoint gives you structured Web Scraping data with computed fields in a single request. No scraping infrastructure to build or maintain.
Example Request
# GET /v1/web/search: Metered: holds 2 credits per call up front and refunds what the call does not use. Cache hits are free and failed calls are refunded.
# Check the price first for 0 credits: add dry_run=1 to the query. Nothing is fetched or charged.
# --max-time 21: this endpoint's measured p99 latency x 1.5.
# --retry 3: retries timeouts, 429 and 5xx, honoring Retry-After.
# -o response.json keeps only the final attempt's body.
# One Idempotency-Key per request, reused on every retry, so a retried call is never charged twice.
IDEMPOTENCY_KEY=$(od -An -N16 -tx1 /dev/urandom | tr -d ' \n')
curl -sS --max-time 21 --retry 3 -o response.json \
-H "x-api-key: YOUR_API_KEY" \
-H "Idempotency-Key: $IDEMPOTENCY_KEY" \
"https://www.socialcrawl.dev/v1/web/search?query=social+media+data+api&limit=10" && cat response.json# GET /v1/web/search: Metered: holds 2 credits per call up front and refunds what the call does not use. Cache hits are free and failed calls are refunded.
# Check the price first for 0 credits: add dry_run=1 to the query. Nothing is fetched or charged.
import os
import random
import time
import uuid
import requests
API_KEY = os.environ.get("SOCIALCRAWL_API_KEY", "YOUR_API_KEY")
TIMEOUT_S = 21 # this endpoint's measured p99 latency x 1.5.
MAX_RETRIES = 3
URL = "https://www.socialcrawl.dev/v1/web/search"
def backoff(attempt):
# Exponential backoff with jitter: about 1 s, 2 s, then 4 s.
return 2 ** attempt * (0.5 + random.random() / 2)
def socialcrawl(params):
# One Idempotency-Key per request, reused on every retry, so a retried call is never charged twice.
idempotency_key = str(uuid.uuid4())
for attempt in range(MAX_RETRIES + 1):
try:
res = requests.get(
URL,
params=params,
headers={"x-api-key": API_KEY, "Idempotency-Key": idempotency_key},
timeout=TIMEOUT_S,
)
body = res.json()
except (requests.RequestException, ValueError):
# A timeout, a dropped connection or a non-JSON gateway page: retry with the same key.
if attempt == MAX_RETRIES:
raise
time.sleep(backoff(attempt))
continue
if body["success"]:
return body
error = body["error"]
# Branch on error["type"], never on the message text.
if error["type"] == "RESOURCE_NOT_FOUND":
return None # refunded: nothing to read
retry_after = res.headers.get("Retry-After")
# SERVICE_UNAVAILABLE without Retry-After is a withdrawn endpoint: permanent.
permanent = error["type"] == "SERVICE_UNAVAILABLE" and retry_after is None
if not error.get("retryable") or permanent or attempt == MAX_RETRIES:
raise RuntimeError(f"{error['type']}: {error['message']} (request_id {body['request_id']})")
# Retry-After (seconds) wins; otherwise back off.
time.sleep(float(retry_after) if retry_after else backoff(attempt))
params = {
"query": "social media data api",
"limit": "10",
}
body = socialcrawl(params)
if body is not None:
print(body["data"])// GET /v1/web/search: Metered: holds 2 credits per call up front and refunds what the call does not use. Cache hits are free and failed calls are refunded.
// Check the price first for 0 credits: add dry_run=1 to the query. Nothing is fetched or charged.
const API_KEY = process.env.SOCIALCRAWL_API_KEY ?? "YOUR_API_KEY";
const TIMEOUT_MS = 21_000; // this endpoint's measured p99 latency x 1.5.
const MAX_RETRIES = 3;
type Page = { next_cursor: string | null; has_more: boolean };
type SocialCrawlResponse =
| { success: true; data: any; pagination?: Page; credits_used: number; request_id: string }
| { success: false; error: { type: string; message: string; retryable: boolean }; request_id: string };
const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
// Exponential backoff with jitter: about 1 s, 2 s, then 4 s.
const backoffMs = (attempt: number) => 2 ** attempt * 1000 * (0.5 + Math.random() / 2);
async function socialcrawl(url: URL) {
// One Idempotency-Key per request, reused on every retry, so a retried call is never charged twice.
const idempotencyKey = crypto.randomUUID();
for (let attempt = 0; ; attempt++) {
let res: Response;
let body: SocialCrawlResponse;
try {
res = await fetch(url, {
headers: {
"x-api-key": API_KEY,
"Idempotency-Key": idempotencyKey,
},
signal: AbortSignal.timeout(TIMEOUT_MS),
});
body = (await res.json()) as SocialCrawlResponse;
} catch (error) {
// A timeout, a dropped connection or a non-JSON gateway page: retry with the same key.
if (attempt >= MAX_RETRIES) throw error;
await sleep(backoffMs(attempt));
continue;
}
if (body.success) return body;
// Branch on error.type, never on the message text.
if (body.error.type === "RESOURCE_NOT_FOUND") return null; // refunded: nothing to read
const retryAfter = res.headers.get("Retry-After");
// SERVICE_UNAVAILABLE without Retry-After is a withdrawn endpoint: permanent.
const permanent = body.error.type === "SERVICE_UNAVAILABLE" && retryAfter === null;
if (!body.error.retryable || permanent || attempt >= MAX_RETRIES) {
throw new Error(`${body.error.type}: ${body.error.message} (request_id ${body.request_id})`);
}
// Retry-After (seconds) wins; otherwise back off.
await sleep(retryAfter ? Number(retryAfter) * 1000 : backoffMs(attempt));
}
}
const url = new URL("https://www.socialcrawl.dev/v1/web/search");
url.searchParams.set("query", "social media data api");
url.searchParams.set("limit", "10");
const body = await socialcrawl(url);
if (body) console.log(body.data);Parameters
| Parameter | Required | Description |
|---|---|---|
| query | Yes | Search query. |
| sources | No | Comma-separated sources: web,news,images. |
| categories | No | CSV of source categories to restrict the search to, e.g. 'github,research'. Omit for an unrestricted web search. |
| limit | No | Results per source, from 1 to 100. |
| country | No | ISO 3166-1 alpha-2 country code the search is run from, e.g. 'US' (default) or 'GB'. Changes which results rank. |
| location | No | Free-text locality to bias results towards, e.g. 'London, United Kingdom'. Finer-grained than country. |
| time_range | No | Recency window as a Google `tbs` value: 'qdr:d' past day, 'qdr:w' past week, 'qdr:m' past month, 'qdr:y' past year. |
| sort_by_date | No | Sort newest-first instead of by relevance. Combines with time_range rather than replacing it. |
| include_domains | No | CSV of domains to restrict results to, e.g. 'nytimes.com,ft.com'. Cannot be combined with exclude_domains - sending both is a 400 before billing. |
| exclude_domains | No | CSV of domains to drop from the results. Cannot be combined with include_domains. |
| include_content | No | Also scrape each result and return its page content as markdown, not just the SERP row. This is what makes the call expensive: it scrapes one page per result, which is why the range runs to 120 credits. |
What does the Web Scraping Search API return?
Every response follows one unified schema. Here is a real, unmodified response body, so you can see the exact fields you get back before spending a credit.
Example response
{
"success": true,
"platform": "web",
"endpoint": "/v1/web/search",
"data": {
"items": [
{
"page": {
"url": "https://docs.brightdata.com/api-reference/scrapers/social-media-apis/overview",
"final_url": "https://docs.brightdata.com/api-reference/scrapers/social-media-apis/overview",
"status_code": null,
"scrape_id": null,
"fetched_at": null,
"title": "Social media scraper APIs - provider Docs",
"description": "## Documentation Index\nThe provider Social Media Scraper APIs collect profiles, posts, reels, jobs, events and comments from 10 social platforms through 67 dedicated endpoints, with one endpoint per data type and input method.",
"source_type": "web",
"published_at": null,
"content": {
"markdown": null,
"html": null,
"raw_html": null,
"summary": null
},
"media": {
"screenshot_url": null,
"audio_url": null,
"video_url": null
},
"extraction": null,
"answer": null,
"highlights": null,
"change_tracking": null,
"page_count": null,
"total_page_count": null,
"fetch": {
"cache_state": null,
"cached_at": null,
"proxy_tier": null
}
}
},
{
"page": {
"url": "https://ensembledata.com/",
"final_url": "https://ensembledata.com/",
"status_code": null,
"scrape_id": null,
"fetched_at": null,
"title": "Social Media Data Scraper APIs",
"description": "# Social Media Scraping API for Data Extraction at Scale\n**Real-time**, **robust** and **scalable** APIs to scrape posts, profiles, comments, hashtags and analytics from social media.\n\n\n\n## Unified API for Social Media Scraping, Data Mining & Analytics\nExtract trending videos, hashtags, and comments from TikTok. Fetch YouTube video metadata, influencer profiles, and campaign analytics.\n\nExplore our [API documentation](https://ensembledata.com/apis/docs) for endpoints covering profile analytics, video data, hashtag discovery, and more.\n\n## Our Social Media Scraper Endpoints\nFetch TikTok profile data including bio, follower counts, and engagement stats for influencer research and social media monitoring.\n\nCollect TikTok posts and video data from any public account for trend research, content audits, and competitive benchmarking.\n\nGet detailed stats on individual TikTok posts including views, shares, hashtags, and engagement data for campaign tracking and reporting.\n\nSearch TikTok by hashtag to find videos, measure reach, and track how campaigns or trending topics are performing on the platform.\n\nCrawl Instagram user profile data including followers, bio, and engagement rates for influencer vetting and brand research.\n\nAccess Instagram Reels data including view counts, play rates, audio tracks, and engagement for short-form video research and influencer analysis.\n\nFetch YouTube channel data including subscriber counts, total views, upload frequency, and metadata for competitive research and video strategy.\n\nAccess YouTube live stream data including concurrent viewers, stream duration, chat volume, and topic metadata for live content research.\n\nExtract YouTube comment data including text, like counts, reply threads, and author info for sentiment analysis and audience research.\n\nRetrieve Threads user profile data including followers, bio, and engagement metrics for influencer research and audience analysis.\n\nExtract Reddit comment threads including top-level replies, nested discussions, upvote scores, and author data for sentiment analysis and community research.\n\nAnalyze Twitch follower data for any streamer including follower lists, growth metrics, and audience insights for influencer research.\n\nRetrieve Twitter user profile data including followers, bio, and engagement stats for social listening, influencer research, and audience analysis.\n\nAccess Twitter tweet histories for any public account including post text, retweet counts, reply totals, and impression data for content and trend analysis.\n\nFetch Snapchat creator profile data including follower counts, engagement rates, and bio information for influencer research and audience analysis.\n\n### Features\nReal-time & Bulk Data API\n\n## 1 Social Data API, Many Use Cases\nPowering businesses with social media data since 2020.\n\n## What is the EnsembleData API designed for?\n### Influencer Discovery & Analytics\nFind and analyze influencers using real-time social media data extraction.\n\n### Social Listening & Intelligence\nMonitor brand conversations, hashtags, and keywords with automated social media scraping. Capture posts, comments, sentiment, and trending topics in real-time for brand monitoring and reputation management.\n\n### Media Intelligence & Research\nScrape Social Data at Scale\n\nCollect public posts, user profiles, engagement metrics, and more for media monitoring, competitive analysis, and AI dataset creation.\n\n### Academic & Market Research\nSocial Data Collection & Analysis\n\nExtract historical social data, user-generated content, and engagement patterns.\n\n## EnsembleData Scraping Guide\n### Frequently Asked Questions\nWho's our social media data scraping API for?\n\nOur social media data API is designed for developers, hedge funds, marketers, researchers, agencies, and brands who need actionable insights from social media platforms. Use cases include market trend analysis, brand monitoring, influencer discovery, competitor analysis, academic research, and powering social media analytics tools. Our API supports integration into SaaS platforms, dashboards, and custom applications for data-driven decision making.\n\nWhich social media platforms do your scraping data APIs support?\n\nAll data is returned as structured, real-time JSON, ideal for analytics, trend tracking, sentiment analysis, and building custom datasets for AI and machine learning.",
"source_type": "web",
"published_at": null,
"content": {
"markdown": null,
"html": null,
"raw_html": null,
"summary": null
},
"media": {
"screenshot_url": null,
"audio_url": null,
"video_url": null
},
"extraction": null,
"answer": null,
"highlights": null,
"change_tracking": null,
"page_count": null,
"total_page_count": null,
"fetch": {
"cache_state": null,
"cached_at": null,
"proxy_tier": null
}
}
}
],
"total": 2,
"dropped": 0
},
"credits_used": 2,
"request_id": "req_example000000",
"cached": false,
"pagination": {
"next_cursor": null,
"has_more": false,
"page_size": 2,
"typical_page_size": null,
"next_page_credits": null
},
"credits_remaining": 9999,
"source": "captured",
"captured_at": "2026-11-02T00:59:19.354Z",
"redacted": true
}Live sample illustrating the unified response shape. Field values reflect the record you query.
How does the Web Scraping Search API work?
Send a GET request with your API key and get back clean, structured JSON in our unified schema. Supported computed fields are populated when the source provides the required inputs.
Method
GET
Response
JSON
How do you scrape social media data in seconds?
The fastest social media scraping API for developers. Scrape profiles, posts, comments, and analytics from 95 platforms covering 10B+ monthly active users.
One schema, every platform
Query 95 platforms with identical response structures. Write your integration once.
Computed fields, not just scraped
When an endpoint supports these metrics and the source provides the required inputs, the normalized record includes engagement_rate, estimated_reach, content_category, and language. Ready to use.
See your data before you code
Visual Data Explorer. Paste any URL, get rich result cards, sortable tables, CSV export.
import requests
response = requests.get(
'https://www.socialcrawl.dev/v1/tiktok/profile',
params={'handle': 'charlidamelio'},
headers={'x-api-key': 'sc_YOUR_API_KEY'}
)
data = response.json(){
"success": true,
"platform": "tiktok",
"data": {
"author": {
"username": "charlidamelio",
"followers": 152400000
},
"engagement": {
"likes": 12400000000,
"engagement_rate": 0.087
},
"metadata": {
"language": "en",
"content_category": "lifestyle"
}
}
}Have a question? We got answers
Find answers to frequently asked questions about SocialCrawl's API, pricing, and capabilities.
Contact usWhat does the Web Search API return?
When should I use the Web Search endpoint?
How much does the Web Search API cost?
What parameters does the Web Search API need?
How do I call the Web Search API?
Ask AI about SocialCrawl
Ready to scrape Web Scraping Search data?
Get your API key and start pulling Web Scraping data in under 60 seconds.
