Web Scraping Scrape API
웹 페이지 하나의 내용을 깔끔한 마크다운이나 HTML로 반환합니다. 최종 URL과 상태 코드, 수집 메타데이터가 함께 오고 원하면 스크린샷까지 붙습니다.
GET /v1/web/scrape2026년 11월 업데이트SocialCrawl 팀이 직접 관리해요
Web Scraping Scrape API, 직접 써 보세요
코드 한 줄 짜기 전에, 실제 데이터부터
Web Scraping Scrape API는 언제 쓰나요
사이트 전체가 대상이라면 crawl 작업을 걸어야 합니다. URL이 이미 정해진 한 페이지를 가져올 때 사용하세요.
API 레퍼런스 영문 원문
Fetches a public web page and returns clean content, metadata, and optional media in the unified WebPage schema. When the page scrapers cannot open a social platform profile or post (a LinkedIn profile, company page or post; an Instagram, Facebook, TikTok, X or Threads profile or post; a Reddit post, subreddit or user page; a YouTube video or channel), the page is read through that platform's own endpoint, for example /v1/linkedin/profile, at that endpoint's price: 5 credits for LinkedIn, 1 for the others. page.native then carries the endpoint (endpoint), what it charged (credits_used) and its full structured record (data). page.content.markdown, html and raw_html are written from that record rather than taken from the site's HTML, and _warnings says so. Only markdown, html and rawHtml requests are read this way. A page the scrapers can read keeps the scrape and its price. When neither page scraper can open an ordinary page, a further source can still read a request for one format, markdown or html; a page read that way carries no screenshot, raw html or page metadata.
95개 플랫폼을 하나의 API로 제공합니다
Scrape API로 무엇을 할 수 있을까요
Scrape 엔드포인트가 통합 스키마와 계산 필드를 담은 Web Scraping 데이터를 한 번의 요청으로 보내드려요. 스크래핑 인프라를 직접 만들거나 유지할 필요가 없어요.
요청 예시
# GET /v1/web/scrape: Metered: holds 1 credit per call up front and refunds what the call does not use. Cache hits are free and failed calls are refunded.
# Check the price first for 0 credits: add dry_run=1 to the query. Nothing is fetched or charged.
# --max-time 25: this endpoint's measured p99 latency x 1.5.
# --retry 3: retries timeouts, 429 and 5xx, honoring Retry-After.
# -o response.json keeps only the final attempt's body.
# One Idempotency-Key per request, reused on every retry, so a retried call is never charged twice.
IDEMPOTENCY_KEY=$(od -An -N16 -tx1 /dev/urandom | tr -d ' \n')
curl -sS --max-time 25 --retry 3 -o response.json \
-H "x-api-key: YOUR_API_KEY" \
-H "Idempotency-Key: $IDEMPOTENCY_KEY" \
"https://www.socialcrawl.dev/v1/web/scrape?url=https%3A%2F%2Fexample.com&formats=markdown" && cat response.json# GET /v1/web/scrape: Metered: holds 1 credit per call up front and refunds what the call does not use. Cache hits are free and failed calls are refunded.
# Check the price first for 0 credits: add dry_run=1 to the query. Nothing is fetched or charged.
import os
import random
import time
import uuid
import requests
API_KEY = os.environ.get("SOCIALCRAWL_API_KEY", "YOUR_API_KEY")
TIMEOUT_S = 25 # this endpoint's measured p99 latency x 1.5.
MAX_RETRIES = 3
URL = "https://www.socialcrawl.dev/v1/web/scrape"
def backoff(attempt):
# Exponential backoff with jitter: about 1 s, 2 s, then 4 s.
return 2 ** attempt * (0.5 + random.random() / 2)
def socialcrawl(params):
# One Idempotency-Key per request, reused on every retry, so a retried call is never charged twice.
idempotency_key = str(uuid.uuid4())
for attempt in range(MAX_RETRIES + 1):
try:
res = requests.get(
URL,
params=params,
headers={"x-api-key": API_KEY, "Idempotency-Key": idempotency_key},
timeout=TIMEOUT_S,
)
body = res.json()
except (requests.RequestException, ValueError):
# A timeout, a dropped connection or a non-JSON gateway page: retry with the same key.
if attempt == MAX_RETRIES:
raise
time.sleep(backoff(attempt))
continue
if body["success"]:
return body
error = body["error"]
# Branch on error["type"], never on the message text.
if error["type"] == "RESOURCE_NOT_FOUND":
return None # refunded: nothing to read
retry_after = res.headers.get("Retry-After")
# SERVICE_UNAVAILABLE without Retry-After is a withdrawn endpoint: permanent.
permanent = error["type"] == "SERVICE_UNAVAILABLE" and retry_after is None
if not error.get("retryable") or permanent or attempt == MAX_RETRIES:
raise RuntimeError(f"{error['type']}: {error['message']} (request_id {body['request_id']})")
# Retry-After (seconds) wins; otherwise back off.
time.sleep(float(retry_after) if retry_after else backoff(attempt))
params = {
"url": "https://example.com",
"formats": "markdown",
}
body = socialcrawl(params)
if body is not None:
print(body["data"])// GET /v1/web/scrape: Metered: holds 1 credit per call up front and refunds what the call does not use. Cache hits are free and failed calls are refunded.
// Check the price first for 0 credits: add dry_run=1 to the query. Nothing is fetched or charged.
const API_KEY = process.env.SOCIALCRAWL_API_KEY ?? "YOUR_API_KEY";
const TIMEOUT_MS = 25_000; // this endpoint's measured p99 latency x 1.5.
const MAX_RETRIES = 3;
type Page = { next_cursor: string | null; has_more: boolean };
type SocialCrawlResponse =
| { success: true; data: any; pagination?: Page; credits_used: number; request_id: string }
| { success: false; error: { type: string; message: string; retryable: boolean }; request_id: string };
const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
// Exponential backoff with jitter: about 1 s, 2 s, then 4 s.
const backoffMs = (attempt: number) => 2 ** attempt * 1000 * (0.5 + Math.random() / 2);
async function socialcrawl(url: URL) {
// One Idempotency-Key per request, reused on every retry, so a retried call is never charged twice.
const idempotencyKey = crypto.randomUUID();
for (let attempt = 0; ; attempt++) {
let res: Response;
let body: SocialCrawlResponse;
try {
res = await fetch(url, {
headers: {
"x-api-key": API_KEY,
"Idempotency-Key": idempotencyKey,
},
signal: AbortSignal.timeout(TIMEOUT_MS),
});
body = (await res.json()) as SocialCrawlResponse;
} catch (error) {
// A timeout, a dropped connection or a non-JSON gateway page: retry with the same key.
if (attempt >= MAX_RETRIES) throw error;
await sleep(backoffMs(attempt));
continue;
}
if (body.success) return body;
// Branch on error.type, never on the message text.
if (body.error.type === "RESOURCE_NOT_FOUND") return null; // refunded: nothing to read
const retryAfter = res.headers.get("Retry-After");
// SERVICE_UNAVAILABLE without Retry-After is a withdrawn endpoint: permanent.
const permanent = body.error.type === "SERVICE_UNAVAILABLE" && retryAfter === null;
if (!body.error.retryable || permanent || attempt >= MAX_RETRIES) {
throw new Error(`${body.error.type}: ${body.error.message} (request_id ${body.request_id})`);
}
// Retry-After (seconds) wins; otherwise back off.
await sleep(retryAfter ? Number(retryAfter) * 1000 : backoffMs(attempt));
}
}
const url = new URL("https://www.socialcrawl.dev/v1/web/scrape");
url.searchParams.set("url", "https://example.com");
url.searchParams.set("formats", "markdown");
const body = await socialcrawl(url);
if (body) console.log(body.data);파라미터
| 파라미터 | 필수 | 설명 |
|---|---|---|
| url | 예 | Public URL to fetch. A Google redirect link (google.com/url?q=...) is fetched as the page it points to; one whose destination Google will not resolve is a 400 INVALID_REQUEST, not charged. |
| formats | 아니오 | Comma-separated output formats such as markdown,screenshot. A social platform URL is read through its own endpoint only for markdown, html and rawHtml. |
| only_main_content | 아니오 | Strip nav, headers, footers and sidebars and keep the article body. Default true; send false when you need the whole page, including the chrome. A markdown-only scrape of a site the primary scraper cannot fetch is served by a second extractor, which returns the whole page. |
| wait_for | 아니오 | Milliseconds to wait after load before capturing, for pages that render their content in JavaScript. Leave unset unless the page comes back empty or half-built. |
| mobile | 아니오 | Render in a mobile viewport with a mobile user agent. Use it when the site serves a different layout to phones. |
| timeout | 아니오 | Hard cap on the page load in milliseconds, 1 to 30000. A page that exceeds it fails and is refunded rather than returning a partial capture. |
| max_age | 아니오 | Accept an upstream-cached copy up to this many milliseconds old, which is faster and still one credit. Default 172800000 (48 hours), or 0 (a live fetch) with `proxy=auto` or `proxy=enhanced`; send 0 to force a live fetch. A cached copy of a page that failed (HTTP 400 or more) is never served: it is fetched again, live. |
| location_country | 아니오 | ISO 3166-1 alpha-2 country code to fetch from, e.g. 'us' or 'de'. Use it for geo-varying pages such as pricing or availability. |
| screenshot_full_page | 아니오 | When a screenshot format is requested, capture the entire scrollable page instead of the visible viewport. |
| include_tags | 아니오 | CSV of CSS selectors or HTML tags to keep, e.g. 'article,main'. Everything outside them is dropped. Use it to pin extraction to a known container. |
| exclude_tags | 아니오 | CSV of CSS selectors or HTML tags to drop, e.g. 'nav,footer,.cookie-banner'. Applied after include_tags. |
| proxy | 아니오 | Proxy tier: basic (the default), auto (a plain request first, then a stronger one if the site blocks it), or enhanced. Use auto when a plain scrape returns a block or CAPTCHA page. auto and enhanced read the page live unless you set max_age. (basic | auto | enhanced) |
| pdf_parse | 아니오 | Parse the target as a PDF and return its extracted text. Send it when the URL points at a PDF rather than an HTML page. |
| block_ads | 아니오 | Block ad and tracker requests, which is faster and quieter. Default true. |
| remove_base64_images | 아니오 | Drop inline base64-encoded images from the returned content instead of carrying them in the payload. Default true. |
Web Scraping Scrape API는 무엇을 돌려주나요
모든 응답은 하나의 통합 스키마를 따라요. 크레딧을 쓰기 전에 어떤 필드가 돌아오는지, 실제 응답 본문 그대로 확인해 보세요.
응답 예시 보기
{
"success": true,
"platform": "web",
"endpoint": "/v1/web/scrape",
"data": {
"page": {
"url": "https://example.com",
"final_url": "https://example.com/",
"status_code": 200,
"scrape_id": "01a0fca4-f04b-7369-9cd6-64e0f4d82214",
"fetched_at": null,
"content": {
"markdown": "Thisdomainisforuseindocumentationexampleswithoutneedingpermission.Thisisnotaservice,avoidrelyingonitfortestingandmonitoringpurposes.\n\nهذا النطاق مُخصص للاستخدام في أمثلة التوثيق دون الحاجة إلى إذن. هذه ليست خدمة، يُرجى تجنب الاعتماد عليها لأغراض الاختبار والمراقبة.\n\n该域名仅用于文档示例,无需获得许可。这并非一项服务,请勿将其用于测试和监控目的。\n\nL’usagedecedomaineestréservéàdesexemplesdedocumentation,sansautorisationpréalable.Ilnes’agitpasd’unservice;sonutilisationàdesfinsdetestoudesurveillanceestàéviter.\n\nДанныйдоменпредназначендляиспользованиявпримерахдокументациибезнеобходимостиполученияпредварительногоразрешения.Этонесервис;нерекомендуетсяегоиспользованиедлятестированияимониторинга.\n\nEstedominioestádestinadoalusoenejemplosdedocumentaciónsinnecesidaddepermiso.Estonoesunservicio,evitarutilizarlopararealizarpruebasomonitoreos.\n\n [Learn more](https://iana.org/help/example-domains)",
"html": null,
"raw_html": null,
"summary": null
},
"media": {
"screenshot_url": null,
"audio_url": null,
"video_url": null
},
"extraction": null,
"answer": null,
"highlights": null,
"change_tracking": null,
"page_count": null,
"total_page_count": null,
"fetch": {
"cache_state": "hit",
"cached_at": "2026-10-02T12:40:55.009Z",
"proxy_tier": null
},
"page_state": {
"value": "content",
"confidence": 0.85,
"skipped_judge": false
}
}
},
"credits_used": 1,
"request_id": "req_example000000",
"cached": false,
"credits_remaining": 9999,
"source": "captured",
"captured_at": "2026-10-02T12:44:23.265Z",
"redacted": true
}통합 응답 구조를 보여 주는 실제 샘플이에요. 필드 값은 조회한 대상에 따라 달라져요.
Web Scraping Scrape API는 어떻게 동작하나요
API 키와 함께 GET 요청을 보내면, 통합 스키마와 계산 필드를 담은 깔끔한 JSON이 돌아와요.
메서드
GET
응답 형식
JSON
소셜 미디어 데이터를 몇 초 만에 수집하는 방법
개발자를 위한 가장 빠른 소셜 미디어 스크래핑 API. 월간 활성 사용자 100억 명 이상을 포괄하는 95개 플랫폼에서 프로필, 게시물, 댓글, 분석 데이터를 수집하세요.
모든 플랫폼을 하나의 스키마로
동일한 응답 구조로 95개 플랫폼을 조회하세요. 연동은 한 번이면 충분합니다.
단순 수집을 넘어 계산된 필드 제공
엔드포인트가 해당 지표를 지원하고 계산에 필요한 원본 값이 있을 때, 정규화된 레코드에 engagement_rate, estimated_reach, content_category, language를 함께 담아 바로 활용할 수 있습니다.
코드 한 줄 쓰기 전에, 데이터부터
Visual Data Explorer에 URL만 붙여넣으면 결과 카드와 정형화된 테이블, CSV 내보내기를 바로 사용할 수 있습니다.
import requests
response = requests.get(
'https://www.socialcrawl.dev/v1/tiktok/profile',
params={'handle': 'charlidamelio'},
headers={'x-api-key': 'sc_YOUR_API_KEY'}
)
data = response.json(){
"success": true,
"platform": "tiktok",
"data": {
"author": {
"username": "charlidamelio",
"followers": 152400000
},
"engagement": {
"likes": 12400000000,
"engagement_rate": 0.087
},
"metadata": {
"language": "en",
"content_category": "lifestyle"
}
}
}자주 묻는 질문
API, 요금제, 기능에 대한 질문과 답변입니다.
문의하기Web Scrape API는 어떤 데이터를 돌려주나요?
Web Scrape 엔드포인트는 언제 쓰면 좋나요?
Web Scrape API 비용은 얼마인가요?
Web Scrape API에는 어떤 파라미터가 필요한가요?
Web Scrape API는 어떻게 호출하나요?
AI에게 SocialCrawl을 물어보세요
Web Scraping Scrape 데이터, 가져올 준비 되셨어요?
API 키 받고 60초 안에 Web Scraping 데이터를 받아 가세요.
