Web Scraping Search API
검색어로 웹과 뉴스, 이미지 결과를 순위대로 찾아 제목과 URL, 요약을 갖춘 하나의 정규화된 목록으로 반환합니다.
GET /v1/web/search2026년 11월 업데이트SocialCrawl 팀이 직접 관리해요
Web Scraping Search API, 직접 써 보세요
코드 한 줄 짜기 전에, 실제 데이터부터
Web Scraping Search API는 언제 쓰나요
URL이 손에 들어온 뒤는 scrape가 맡습니다. 어떤 URL이 필요한지 아직 모를 때 사용하세요.
API 레퍼런스 영문 원문
Searches web, news, and image sources and returns a single normalized list of web page results. When the primary index cannot answer, a second web index serves a web or news search in the same shape and at the same price: url, title, description, source_type and the news publish time are filled, rows are ranked by that index, and with sort_by_date the news rows are ordered newest first. It does not serve images, categories, location, include_content, more than 20 results a source or a time_range other than qdr:d, qdr:w, qdr:m and qdr:y.
95개 플랫폼을 하나의 API로 제공합니다
Search API로 무엇을 할 수 있을까요
Search 엔드포인트가 통합 스키마와 계산 필드를 담은 Web Scraping 데이터를 한 번의 요청으로 보내드려요. 스크래핑 인프라를 직접 만들거나 유지할 필요가 없어요.
요청 예시
# GET /v1/web/search: Metered: holds 2 credits per call up front and refunds what the call does not use. Cache hits are free and failed calls are refunded.
# Check the price first for 0 credits: add dry_run=1 to the query. Nothing is fetched or charged.
# --max-time 21: this endpoint's measured p99 latency x 1.5.
# --retry 3: retries timeouts, 429 and 5xx, honoring Retry-After.
# -o response.json keeps only the final attempt's body.
# One Idempotency-Key per request, reused on every retry, so a retried call is never charged twice.
IDEMPOTENCY_KEY=$(od -An -N16 -tx1 /dev/urandom | tr -d ' \n')
curl -sS --max-time 21 --retry 3 -o response.json \
-H "x-api-key: YOUR_API_KEY" \
-H "Idempotency-Key: $IDEMPOTENCY_KEY" \
"https://www.socialcrawl.dev/v1/web/search?query=social+media+data+api&limit=10" && cat response.json# GET /v1/web/search: Metered: holds 2 credits per call up front and refunds what the call does not use. Cache hits are free and failed calls are refunded.
# Check the price first for 0 credits: add dry_run=1 to the query. Nothing is fetched or charged.
import os
import random
import time
import uuid
import requests
API_KEY = os.environ.get("SOCIALCRAWL_API_KEY", "YOUR_API_KEY")
TIMEOUT_S = 21 # this endpoint's measured p99 latency x 1.5.
MAX_RETRIES = 3
URL = "https://www.socialcrawl.dev/v1/web/search"
def backoff(attempt):
# Exponential backoff with jitter: about 1 s, 2 s, then 4 s.
return 2 ** attempt * (0.5 + random.random() / 2)
def socialcrawl(params):
# One Idempotency-Key per request, reused on every retry, so a retried call is never charged twice.
idempotency_key = str(uuid.uuid4())
for attempt in range(MAX_RETRIES + 1):
try:
res = requests.get(
URL,
params=params,
headers={"x-api-key": API_KEY, "Idempotency-Key": idempotency_key},
timeout=TIMEOUT_S,
)
body = res.json()
except (requests.RequestException, ValueError):
# A timeout, a dropped connection or a non-JSON gateway page: retry with the same key.
if attempt == MAX_RETRIES:
raise
time.sleep(backoff(attempt))
continue
if body["success"]:
return body
error = body["error"]
# Branch on error["type"], never on the message text.
if error["type"] == "RESOURCE_NOT_FOUND":
return None # refunded: nothing to read
retry_after = res.headers.get("Retry-After")
# SERVICE_UNAVAILABLE without Retry-After is a withdrawn endpoint: permanent.
permanent = error["type"] == "SERVICE_UNAVAILABLE" and retry_after is None
if not error.get("retryable") or permanent or attempt == MAX_RETRIES:
raise RuntimeError(f"{error['type']}: {error['message']} (request_id {body['request_id']})")
# Retry-After (seconds) wins; otherwise back off.
time.sleep(float(retry_after) if retry_after else backoff(attempt))
params = {
"query": "social media data api",
"limit": "10",
}
body = socialcrawl(params)
if body is not None:
print(body["data"])// GET /v1/web/search: Metered: holds 2 credits per call up front and refunds what the call does not use. Cache hits are free and failed calls are refunded.
// Check the price first for 0 credits: add dry_run=1 to the query. Nothing is fetched or charged.
const API_KEY = process.env.SOCIALCRAWL_API_KEY ?? "YOUR_API_KEY";
const TIMEOUT_MS = 21_000; // this endpoint's measured p99 latency x 1.5.
const MAX_RETRIES = 3;
type Page = { next_cursor: string | null; has_more: boolean };
type SocialCrawlResponse =
| { success: true; data: any; pagination?: Page; credits_used: number; request_id: string }
| { success: false; error: { type: string; message: string; retryable: boolean }; request_id: string };
const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
// Exponential backoff with jitter: about 1 s, 2 s, then 4 s.
const backoffMs = (attempt: number) => 2 ** attempt * 1000 * (0.5 + Math.random() / 2);
async function socialcrawl(url: URL) {
// One Idempotency-Key per request, reused on every retry, so a retried call is never charged twice.
const idempotencyKey = crypto.randomUUID();
for (let attempt = 0; ; attempt++) {
let res: Response;
let body: SocialCrawlResponse;
try {
res = await fetch(url, {
headers: {
"x-api-key": API_KEY,
"Idempotency-Key": idempotencyKey,
},
signal: AbortSignal.timeout(TIMEOUT_MS),
});
body = (await res.json()) as SocialCrawlResponse;
} catch (error) {
// A timeout, a dropped connection or a non-JSON gateway page: retry with the same key.
if (attempt >= MAX_RETRIES) throw error;
await sleep(backoffMs(attempt));
continue;
}
if (body.success) return body;
// Branch on error.type, never on the message text.
if (body.error.type === "RESOURCE_NOT_FOUND") return null; // refunded: nothing to read
const retryAfter = res.headers.get("Retry-After");
// SERVICE_UNAVAILABLE without Retry-After is a withdrawn endpoint: permanent.
const permanent = body.error.type === "SERVICE_UNAVAILABLE" && retryAfter === null;
if (!body.error.retryable || permanent || attempt >= MAX_RETRIES) {
throw new Error(`${body.error.type}: ${body.error.message} (request_id ${body.request_id})`);
}
// Retry-After (seconds) wins; otherwise back off.
await sleep(retryAfter ? Number(retryAfter) * 1000 : backoffMs(attempt));
}
}
const url = new URL("https://www.socialcrawl.dev/v1/web/search");
url.searchParams.set("query", "social media data api");
url.searchParams.set("limit", "10");
const body = await socialcrawl(url);
if (body) console.log(body.data);파라미터
| 파라미터 | 필수 | 설명 |
|---|---|---|
| query | 예 | Search query. |
| sources | 아니오 | Comma-separated sources: web,news,images. |
| categories | 아니오 | CSV of source categories to restrict the search to, e.g. 'github,research'. Omit for an unrestricted web search. |
| limit | 아니오 | Results per source, from 1 to 100. |
| country | 아니오 | ISO 3166-1 alpha-2 country code the search is run from, e.g. 'US' (default) or 'GB'. Changes which results rank. |
| location | 아니오 | Free-text locality to bias results towards, e.g. 'London, United Kingdom'. Finer-grained than country. |
| time_range | 아니오 | Recency window as a Google `tbs` value: 'qdr:d' past day, 'qdr:w' past week, 'qdr:m' past month, 'qdr:y' past year. |
| sort_by_date | 아니오 | Sort newest-first instead of by relevance. Combines with time_range rather than replacing it. |
| include_domains | 아니오 | CSV of domains to restrict results to, e.g. 'nytimes.com,ft.com'. Cannot be combined with exclude_domains - sending both is a 400 before billing. |
| exclude_domains | 아니오 | CSV of domains to drop from the results. Cannot be combined with include_domains. |
| include_content | 아니오 | Also scrape each result and return its page content as markdown, not just the SERP row. This is what makes the call expensive: it scrapes one page per result, which is why the range runs to 120 credits. |
Web Scraping Search API는 무엇을 돌려주나요
모든 응답은 하나의 통합 스키마를 따라요. 크레딧을 쓰기 전에 어떤 필드가 돌아오는지, 실제 응답 본문 그대로 확인해 보세요.
응답 예시 보기
{
"success": true,
"platform": "web",
"endpoint": "/v1/web/search",
"data": {
"items": [
{
"page": {
"url": "https://docs.brightdata.com/api-reference/scrapers/social-media-apis/overview",
"final_url": "https://docs.brightdata.com/api-reference/scrapers/social-media-apis/overview",
"status_code": null,
"scrape_id": null,
"fetched_at": null,
"title": "Social media scraper APIs - provider Docs",
"description": "## Documentation Index\nThe provider Social Media Scraper APIs collect profiles, posts, reels, jobs, events and comments from 10 social platforms through 67 dedicated endpoints, with one endpoint per data type and input method.",
"source_type": "web",
"published_at": null,
"content": {
"markdown": null,
"html": null,
"raw_html": null,
"summary": null
},
"media": {
"screenshot_url": null,
"audio_url": null,
"video_url": null
},
"extraction": null,
"answer": null,
"highlights": null,
"change_tracking": null,
"page_count": null,
"total_page_count": null,
"fetch": {
"cache_state": null,
"cached_at": null,
"proxy_tier": null
}
}
},
{
"page": {
"url": "https://ensembledata.com/",
"final_url": "https://ensembledata.com/",
"status_code": null,
"scrape_id": null,
"fetched_at": null,
"title": "Social Media Data Scraper APIs",
"description": "# Social Media Scraping API for Data Extraction at Scale\n**Real-time**, **robust** and **scalable** APIs to scrape posts, profiles, comments, hashtags and analytics from social media.\n\n\n\n## Unified API for Social Media Scraping, Data Mining & Analytics\nExtract trending videos, hashtags, and comments from TikTok. Fetch YouTube video metadata, influencer profiles, and campaign analytics.\n\nExplore our [API documentation](https://ensembledata.com/apis/docs) for endpoints covering profile analytics, video data, hashtag discovery, and more.\n\n## Our Social Media Scraper Endpoints\nFetch TikTok profile data including bio, follower counts, and engagement stats for influencer research and social media monitoring.\n\nCollect TikTok posts and video data from any public account for trend research, content audits, and competitive benchmarking.\n\nGet detailed stats on individual TikTok posts including views, shares, hashtags, and engagement data for campaign tracking and reporting.\n\nSearch TikTok by hashtag to find videos, measure reach, and track how campaigns or trending topics are performing on the platform.\n\nCrawl Instagram user profile data including followers, bio, and engagement rates for influencer vetting and brand research.\n\nAccess Instagram Reels data including view counts, play rates, audio tracks, and engagement for short-form video research and influencer analysis.\n\nFetch YouTube channel data including subscriber counts, total views, upload frequency, and metadata for competitive research and video strategy.\n\nAccess YouTube live stream data including concurrent viewers, stream duration, chat volume, and topic metadata for live content research.\n\nExtract YouTube comment data including text, like counts, reply threads, and author info for sentiment analysis and audience research.\n\nRetrieve Threads user profile data including followers, bio, and engagement metrics for influencer research and audience analysis.\n\nExtract Reddit comment threads including top-level replies, nested discussions, upvote scores, and author data for sentiment analysis and community research.\n\nAnalyze Twitch follower data for any streamer including follower lists, growth metrics, and audience insights for influencer research.\n\nRetrieve Twitter user profile data including followers, bio, and engagement stats for social listening, influencer research, and audience analysis.\n\nAccess Twitter tweet histories for any public account including post text, retweet counts, reply totals, and impression data for content and trend analysis.\n\nFetch Snapchat creator profile data including follower counts, engagement rates, and bio information for influencer research and audience analysis.\n\n### Features\nReal-time & Bulk Data API\n\n## 1 Social Data API, Many Use Cases\nPowering businesses with social media data since 2020.\n\n## What is the EnsembleData API designed for?\n### Influencer Discovery & Analytics\nFind and analyze influencers using real-time social media data extraction.\n\n### Social Listening & Intelligence\nMonitor brand conversations, hashtags, and keywords with automated social media scraping. Capture posts, comments, sentiment, and trending topics in real-time for brand monitoring and reputation management.\n\n### Media Intelligence & Research\nScrape Social Data at Scale\n\nCollect public posts, user profiles, engagement metrics, and more for media monitoring, competitive analysis, and AI dataset creation.\n\n### Academic & Market Research\nSocial Data Collection & Analysis\n\nExtract historical social data, user-generated content, and engagement patterns.\n\n## EnsembleData Scraping Guide\n### Frequently Asked Questions\nWho's our social media data scraping API for?\n\nOur social media data API is designed for developers, hedge funds, marketers, researchers, agencies, and brands who need actionable insights from social media platforms. Use cases include market trend analysis, brand monitoring, influencer discovery, competitor analysis, academic research, and powering social media analytics tools. Our API supports integration into SaaS platforms, dashboards, and custom applications for data-driven decision making.\n\nWhich social media platforms do your scraping data APIs support?\n\nAll data is returned as structured, real-time JSON, ideal for analytics, trend tracking, sentiment analysis, and building custom datasets for AI and machine learning.",
"source_type": "web",
"published_at": null,
"content": {
"markdown": null,
"html": null,
"raw_html": null,
"summary": null
},
"media": {
"screenshot_url": null,
"audio_url": null,
"video_url": null
},
"extraction": null,
"answer": null,
"highlights": null,
"change_tracking": null,
"page_count": null,
"total_page_count": null,
"fetch": {
"cache_state": null,
"cached_at": null,
"proxy_tier": null
}
}
}
],
"total": 2,
"dropped": 0
},
"credits_used": 2,
"request_id": "req_example000000",
"cached": false,
"pagination": {
"next_cursor": null,
"has_more": false,
"page_size": 2,
"typical_page_size": null,
"next_page_credits": null
},
"credits_remaining": 9999,
"source": "captured",
"captured_at": "2026-11-02T00:59:19.354Z",
"redacted": true
}통합 응답 구조를 보여 주는 실제 샘플이에요. 필드 값은 조회한 대상에 따라 달라져요.
Web Scraping Search API는 어떻게 동작하나요
API 키와 함께 GET 요청을 보내면, 통합 스키마와 계산 필드를 담은 깔끔한 JSON이 돌아와요.
메서드
GET
응답 형식
JSON
소셜 미디어 데이터를 몇 초 만에 수집하는 방법
개발자를 위한 가장 빠른 소셜 미디어 스크래핑 API. 월간 활성 사용자 100억 명 이상을 포괄하는 95개 플랫폼에서 프로필, 게시물, 댓글, 분석 데이터를 수집하세요.
모든 플랫폼을 하나의 스키마로
동일한 응답 구조로 95개 플랫폼을 조회하세요. 연동은 한 번이면 충분합니다.
단순 수집을 넘어 계산된 필드 제공
엔드포인트가 해당 지표를 지원하고 계산에 필요한 원본 값이 있을 때, 정규화된 레코드에 engagement_rate, estimated_reach, content_category, language를 함께 담아 바로 활용할 수 있습니다.
코드 한 줄 쓰기 전에, 데이터부터
Visual Data Explorer에 URL만 붙여넣으면 결과 카드와 정형화된 테이블, CSV 내보내기를 바로 사용할 수 있습니다.
import requests
response = requests.get(
'https://www.socialcrawl.dev/v1/tiktok/profile',
params={'handle': 'charlidamelio'},
headers={'x-api-key': 'sc_YOUR_API_KEY'}
)
data = response.json(){
"success": true,
"platform": "tiktok",
"data": {
"author": {
"username": "charlidamelio",
"followers": 152400000
},
"engagement": {
"likes": 12400000000,
"engagement_rate": 0.087
},
"metadata": {
"language": "en",
"content_category": "lifestyle"
}
}
}자주 묻는 질문
API, 요금제, 기능에 대한 질문과 답변입니다.
문의하기Web Search API는 어떤 데이터를 돌려주나요?
Web Search 엔드포인트는 언제 쓰면 좋나요?
Web Search API 비용은 얼마인가요?
Web Search API에는 어떤 파라미터가 필요한가요?
Web Search API는 어떻게 호출하나요?
AI에게 SocialCrawl을 물어보세요
Web Scraping Search 데이터, 가져올 준비 되셨어요?
API 키 받고 60초 안에 Web Scraping 데이터를 받아 가세요.
