100 크레딧 무료, 카드 등록 없이지금 시작하기
Logo
Web Scraping logo
100 크레딧 무료, 카드 등록 없이

Web Scraping Scrape API

웹 페이지 하나의 내용을 깔끔한 마크다운이나 HTML로 반환합니다. 최종 URL과 상태 코드, 수집 메타데이터가 함께 오고 원하면 스크린샷까지 붙습니다.

GET /v1/web/scrape

2026년 11월 업데이트SocialCrawl 팀이 직접 관리해요

Web Scraping Scrape API, 직접 써 보세요

코드 한 줄 짜기 전에, 실제 데이터부터

GET/v1/web/scrape

Public URL to fetch. A Google redirect link (google.com/url?q=...) is fetched as the page it points to; one whose destination Google will not resolve is a 400 INVALID_REQUEST, not charged.

선택 파라미터 14개

Comma-separated output formats such as markdown,screenshot. A social platform URL is read through its own endpoint only for markdown, html and rawHtml.

Strip nav, headers, footers and sidebars and keep the article body. Default true; send false when you need the whole page, including the chrome. A markdown-only scrape of a site the primary scraper cannot fetch is served by a second extractor, which returns the whole page.

Milliseconds to wait after load before capturing, for pages that render their content in JavaScript. Leave unset unless the page comes back empty or half-built.

Render in a mobile viewport with a mobile user agent. Use it when the site serves a different layout to phones.

Hard cap on the page load in milliseconds, 1 to 30000. A page that exceeds it fails and is refunded rather than returning a partial capture.

Accept an upstream-cached copy up to this many milliseconds old, which is faster and still one credit. Default 172800000 (48 hours), or 0 (a live fetch) with `proxy=auto` or `proxy=enhanced`; send 0 to force a live fetch. A cached copy of a page that failed (HTTP 400 or more) is never served: it is fetched again, live.

ISO 3166-1 alpha-2 country code to fetch from, e.g. 'us' or 'de'. Use it for geo-varying pages such as pricing or availability.

When a screenshot format is requested, capture the entire scrollable page instead of the visible viewport.

CSV of CSS selectors or HTML tags to keep, e.g. 'article,main'. Everything outside them is dropped. Use it to pin extraction to a known container.

CSV of CSS selectors or HTML tags to drop, e.g. 'nav,footer,.cookie-banner'. Applied after include_tags.

Proxy tier: basic (the default), auto (a plain request first, then a stronger one if the site blocks it), or enhanced. Use auto when a plain scrape returns a block or CAPTCHA page. auto and enhanced read the page live unless you set max_age.

Parse the target as a PDF and return its extracted text. Send it when the URL points at a PDF rather than an HTML page.

Block ad and tracker requests, which is faster and quieter. Default true.

Drop inline base64-encoded images from the returned content instead of carrying them in the payload. Default true.

Web Scraping Scrape API는 언제 쓰나요

사이트 전체가 대상이라면 crawl 작업을 걸어야 합니다. URL이 이미 정해진 한 페이지를 가져올 때 사용하세요.

API 레퍼런스 영문 원문

Fetches a public web page and returns clean content, metadata, and optional media in the unified WebPage schema. When the page scrapers cannot open a social platform profile or post (a LinkedIn profile, company page or post; an Instagram, Facebook, TikTok, X or Threads profile or post; a Reddit post, subreddit or user page; a YouTube video or channel), the page is read through that platform's own endpoint, for example /v1/linkedin/profile, at that endpoint's price: 5 credits for LinkedIn, 1 for the others. page.native then carries the endpoint (endpoint), what it charged (credits_used) and its full structured record (data). page.content.markdown, html and raw_html are written from that record rather than taken from the site's HTML, and _warnings says so. Only markdown, html and rawHtml requests are read this way. A page the scrapers can read keeps the scrape and its price. When neither page scraper can open an ordinary page, a further source can still read a request for one format, markdown or html; a page read that way carries no screenshot, raw html or page metadata.

95개 플랫폼을 하나의 API로 제공합니다

·TikTok logoTikTok·Instagram logoInstagram·YouTube logoYouTube·Facebook logoFacebook·X logoX·LinkedIn logoLinkedIn·Reddit logoReddit·Threads logoThreads·Pinterest logoPinterest·Twitch logoTwitch·Truth Social logoTruth Social·Snapchat logoSnapchat·Kick logoKick·Bluesky logoBluesky·Kwai logoKwai·Rumble logoRumble·Spotify logoSpotify·Apple Music logoApple Music·TikTok Shop logoTikTok Shop·Amazon Shop logoAmazon Shop·Google Shopping logoGoogle Shopping·Trustpilot logoTrustpilot·TripAdvisor logoTripAdvisor·Yelp logoYelp·Linktree logoLinktree·Komi logoKomi·Pillar logoPillar·lnk.bio logolnk.bio·Facebook Ads logoFacebook Ads·Google Ads logoGoogle Ads·LinkedIn Ads logoLinkedIn Ads·Google Search logoGoogle Search·Google News logoGoogle News·Finance logoFinance·Polymarket logoPolymarket·Tavily logoTavily·Hacker News logoHacker News·GitHub logoGitHub·Perplexity logoPerplexity·Naver logoNaver·Utility logoUtility·Universal Search logoUniversal Search
·TikTok logoTikTok·Instagram logoInstagram·YouTube logoYouTube·Facebook logoFacebook·X logoX·LinkedIn logoLinkedIn·Reddit logoReddit·Threads logoThreads·Pinterest logoPinterest·Twitch logoTwitch·Truth Social logoTruth Social·Snapchat logoSnapchat·Kick logoKick·Bluesky logoBluesky·Kwai logoKwai·Rumble logoRumble·Spotify logoSpotify·Apple Music logoApple Music·TikTok Shop logoTikTok Shop·Amazon Shop logoAmazon Shop·Google Shopping logoGoogle Shopping·Trustpilot logoTrustpilot·TripAdvisor logoTripAdvisor·Yelp logoYelp·Linktree logoLinktree·Komi logoKomi·Pillar logoPillar·lnk.bio logolnk.bio·Facebook Ads logoFacebook Ads·Google Ads logoGoogle Ads·LinkedIn Ads logoLinkedIn Ads·Google Search logoGoogle Search·Google News logoGoogle News·Finance logoFinance·Polymarket logoPolymarket·Tavily logoTavily·Hacker News logoHacker News·GitHub logoGitHub·Perplexity logoPerplexity·Naver logoNaver·Utility logoUtility·Universal Search logoUniversal Search
·TikTok logoTikTok·Instagram logoInstagram·YouTube logoYouTube·Facebook logoFacebook·X logoX·LinkedIn logoLinkedIn·Reddit logoReddit·Threads logoThreads·Pinterest logoPinterest·Twitch logoTwitch·Truth Social logoTruth Social·Snapchat logoSnapchat·Kick logoKick·Bluesky logoBluesky·Kwai logoKwai·Rumble logoRumble·Spotify logoSpotify·Apple Music logoApple Music·TikTok Shop logoTikTok Shop·Amazon Shop logoAmazon Shop·Google Shopping logoGoogle Shopping·Trustpilot logoTrustpilot·TripAdvisor logoTripAdvisor·Yelp logoYelp·Linktree logoLinktree·Komi logoKomi·Pillar logoPillar·lnk.bio logolnk.bio·Facebook Ads logoFacebook Ads·Google Ads logoGoogle Ads·LinkedIn Ads logoLinkedIn Ads·Google Search logoGoogle Search·Google News logoGoogle News·Finance logoFinance·Polymarket logoPolymarket·Tavily logoTavily·Hacker News logoHacker News·GitHub logoGitHub·Perplexity logoPerplexity·Naver logoNaver·Utility logoUtility·Universal Search logoUniversal Search
·TikTok logoTikTok·Instagram logoInstagram·YouTube logoYouTube·Facebook logoFacebook·X logoX·LinkedIn logoLinkedIn·Reddit logoReddit·Threads logoThreads·Pinterest logoPinterest·Twitch logoTwitch·Truth Social logoTruth Social·Snapchat logoSnapchat·Kick logoKick·Bluesky logoBluesky·Kwai logoKwai·Rumble logoRumble·Spotify logoSpotify·Apple Music logoApple Music·TikTok Shop logoTikTok Shop·Amazon Shop logoAmazon Shop·Google Shopping logoGoogle Shopping·Trustpilot logoTrustpilot·TripAdvisor logoTripAdvisor·Yelp logoYelp·Linktree logoLinktree·Komi logoKomi·Pillar logoPillar·lnk.bio logolnk.bio·Facebook Ads logoFacebook Ads·Google Ads logoGoogle Ads·LinkedIn Ads logoLinkedIn Ads·Google Search logoGoogle Search·Google News logoGoogle News·Finance logoFinance·Polymarket logoPolymarket·Tavily logoTavily·Hacker News logoHacker News·GitHub logoGitHub·Perplexity logoPerplexity·Naver logoNaver·Utility logoUtility·Universal Search logoUniversal Search
Web Scraping API

Scrape API로 무엇을 할 수 있을까요

Scrape 엔드포인트가 통합 스키마와 계산 필드를 담은 Web Scraping 데이터를 한 번의 요청으로 보내드려요. 스크래핑 인프라를 직접 만들거나 유지할 필요가 없어요.

요청 예시

# GET /v1/web/scrape: Metered: holds 1 credit per call up front and refunds what the call does not use. Cache hits are free and failed calls are refunded.
# Check the price first for 0 credits: add dry_run=1 to the query. Nothing is fetched or charged.
# --max-time 25: this endpoint's measured p99 latency x 1.5.
# --retry 3: retries timeouts, 429 and 5xx, honoring Retry-After.
# -o response.json keeps only the final attempt's body.
# One Idempotency-Key per request, reused on every retry, so a retried call is never charged twice.
IDEMPOTENCY_KEY=$(od -An -N16 -tx1 /dev/urandom | tr -d ' \n')
curl -sS --max-time 25 --retry 3 -o response.json \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Idempotency-Key: $IDEMPOTENCY_KEY" \
  "https://www.socialcrawl.dev/v1/web/scrape?url=https%3A%2F%2Fexample.com&formats=markdown" && cat response.json
# GET /v1/web/scrape: Metered: holds 1 credit per call up front and refunds what the call does not use. Cache hits are free and failed calls are refunded.
# Check the price first for 0 credits: add dry_run=1 to the query. Nothing is fetched or charged.
import os
import random
import time
import uuid

import requests

API_KEY = os.environ.get("SOCIALCRAWL_API_KEY", "YOUR_API_KEY")
TIMEOUT_S = 25  # this endpoint's measured p99 latency x 1.5.
MAX_RETRIES = 3
URL = "https://www.socialcrawl.dev/v1/web/scrape"


def backoff(attempt):
    # Exponential backoff with jitter: about 1 s, 2 s, then 4 s.
    return 2 ** attempt * (0.5 + random.random() / 2)


def socialcrawl(params):
    # One Idempotency-Key per request, reused on every retry, so a retried call is never charged twice.
    idempotency_key = str(uuid.uuid4())
    for attempt in range(MAX_RETRIES + 1):
        try:
            res = requests.get(
                URL,
                params=params,
                headers={"x-api-key": API_KEY, "Idempotency-Key": idempotency_key},
                timeout=TIMEOUT_S,
            )
            body = res.json()
        except (requests.RequestException, ValueError):
            # A timeout, a dropped connection or a non-JSON gateway page: retry with the same key.
            if attempt == MAX_RETRIES:
                raise
            time.sleep(backoff(attempt))
            continue
        if body["success"]:
            return body
        error = body["error"]
        # Branch on error["type"], never on the message text.
        if error["type"] == "RESOURCE_NOT_FOUND":
            return None  # refunded: nothing to read
        retry_after = res.headers.get("Retry-After")
        # SERVICE_UNAVAILABLE without Retry-After is a withdrawn endpoint: permanent.
        permanent = error["type"] == "SERVICE_UNAVAILABLE" and retry_after is None
        if not error.get("retryable") or permanent or attempt == MAX_RETRIES:
            raise RuntimeError(f"{error['type']}: {error['message']} (request_id {body['request_id']})")
        # Retry-After (seconds) wins; otherwise back off.
        time.sleep(float(retry_after) if retry_after else backoff(attempt))


params = {
    "url": "https://example.com",
    "formats": "markdown",
}
body = socialcrawl(params)
if body is not None:
    print(body["data"])
// GET /v1/web/scrape: Metered: holds 1 credit per call up front and refunds what the call does not use. Cache hits are free and failed calls are refunded.
// Check the price first for 0 credits: add dry_run=1 to the query. Nothing is fetched or charged.
const API_KEY = process.env.SOCIALCRAWL_API_KEY ?? "YOUR_API_KEY";
const TIMEOUT_MS = 25_000; // this endpoint's measured p99 latency x 1.5.
const MAX_RETRIES = 3;

type Page = { next_cursor: string | null; has_more: boolean };
type SocialCrawlResponse =
  | { success: true; data: any; pagination?: Page; credits_used: number; request_id: string }
  | { success: false; error: { type: string; message: string; retryable: boolean }; request_id: string };

const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
// Exponential backoff with jitter: about 1 s, 2 s, then 4 s.
const backoffMs = (attempt: number) => 2 ** attempt * 1000 * (0.5 + Math.random() / 2);

async function socialcrawl(url: URL) {
  // One Idempotency-Key per request, reused on every retry, so a retried call is never charged twice.
  const idempotencyKey = crypto.randomUUID();
  for (let attempt = 0; ; attempt++) {
    let res: Response;
    let body: SocialCrawlResponse;
    try {
      res = await fetch(url, {
        headers: {
          "x-api-key": API_KEY,
          "Idempotency-Key": idempotencyKey,
        },
        signal: AbortSignal.timeout(TIMEOUT_MS),
      });
      body = (await res.json()) as SocialCrawlResponse;
    } catch (error) {
      // A timeout, a dropped connection or a non-JSON gateway page: retry with the same key.
      if (attempt >= MAX_RETRIES) throw error;
      await sleep(backoffMs(attempt));
      continue;
    }
    if (body.success) return body;
    // Branch on error.type, never on the message text.
    if (body.error.type === "RESOURCE_NOT_FOUND") return null; // refunded: nothing to read
    const retryAfter = res.headers.get("Retry-After");
    // SERVICE_UNAVAILABLE without Retry-After is a withdrawn endpoint: permanent.
    const permanent = body.error.type === "SERVICE_UNAVAILABLE" && retryAfter === null;
    if (!body.error.retryable || permanent || attempt >= MAX_RETRIES) {
      throw new Error(`${body.error.type}: ${body.error.message} (request_id ${body.request_id})`);
    }
    // Retry-After (seconds) wins; otherwise back off.
    await sleep(retryAfter ? Number(retryAfter) * 1000 : backoffMs(attempt));
  }
}

const url = new URL("https://www.socialcrawl.dev/v1/web/scrape");
url.searchParams.set("url", "https://example.com");
url.searchParams.set("formats", "markdown");

const body = await socialcrawl(url);
if (body) console.log(body.data);

파라미터

파라미터필수설명
url예Public URL to fetch. A Google redirect link (google.com/url?q=...) is fetched as the page it points to; one whose destination Google will not resolve is a 400 INVALID_REQUEST, not charged.
formats아니오Comma-separated output formats such as markdown,screenshot. A social platform URL is read through its own endpoint only for markdown, html and rawHtml.
only_main_content아니오Strip nav, headers, footers and sidebars and keep the article body. Default true; send false when you need the whole page, including the chrome. A markdown-only scrape of a site the primary scraper cannot fetch is served by a second extractor, which returns the whole page.
wait_for아니오Milliseconds to wait after load before capturing, for pages that render their content in JavaScript. Leave unset unless the page comes back empty or half-built.
mobile아니오Render in a mobile viewport with a mobile user agent. Use it when the site serves a different layout to phones.
timeout아니오Hard cap on the page load in milliseconds, 1 to 30000. A page that exceeds it fails and is refunded rather than returning a partial capture.
max_age아니오Accept an upstream-cached copy up to this many milliseconds old, which is faster and still one credit. Default 172800000 (48 hours), or 0 (a live fetch) with `proxy=auto` or `proxy=enhanced`; send 0 to force a live fetch. A cached copy of a page that failed (HTTP 400 or more) is never served: it is fetched again, live.
location_country아니오ISO 3166-1 alpha-2 country code to fetch from, e.g. 'us' or 'de'. Use it for geo-varying pages such as pricing or availability.
screenshot_full_page아니오When a screenshot format is requested, capture the entire scrollable page instead of the visible viewport.
include_tags아니오CSV of CSS selectors or HTML tags to keep, e.g. 'article,main'. Everything outside them is dropped. Use it to pin extraction to a known container.
exclude_tags아니오CSV of CSS selectors or HTML tags to drop, e.g. 'nav,footer,.cookie-banner'. Applied after include_tags.
proxy아니오Proxy tier: basic (the default), auto (a plain request first, then a stronger one if the site blocks it), or enhanced. Use auto when a plain scrape returns a block or CAPTCHA page. auto and enhanced read the page live unless you set max_age. (basic | auto | enhanced)
pdf_parse아니오Parse the target as a PDF and return its extracted text. Send it when the URL points at a PDF rather than an HTML page.
block_ads아니오Block ad and tracker requests, which is faster and quieter. Default true.
remove_base64_images아니오Drop inline base64-encoded images from the returned content instead of carrying them in the payload. Default true.
응답 예시

Web Scraping Scrape API는 무엇을 돌려주나요

모든 응답은 하나의 통합 스키마를 따라요. 크레딧을 쓰기 전에 어떤 필드가 돌아오는지, 실제 응답 본문 그대로 확인해 보세요.

응답 예시 보기
{
  "success": true,
  "platform": "web",
  "endpoint": "/v1/web/scrape",
  "data": {
    "page": {
      "url": "https://example.com",
      "final_url": "https://example.com/",
      "status_code": 200,
      "scrape_id": "01a0fca4-f04b-7369-9cd6-64e0f4d82214",
      "fetched_at": null,
      "content": {
        "markdown": "Thisdomainisforuseindocumentationexampleswithoutneedingpermission.Thisisnotaservice,avoidrelyingonitfortestingandmonitoringpurposes.\n\nهذا النطاق مُخصص للاستخدام في أمثلة التوثيق دون الحاجة إلى إذن. هذه ليست خدمة، يُرجى تجنب الاعتماد عليها لأغراض الاختبار والمراقبة.\n\n该域名仅用于文档示例,无需获得许可。这并非一项服务,请勿将其用于测试和监控目的。\n\nL’usagedecedomaineestréservéàdesexemplesdedocumentation,sansautorisationpréalable.Ilnes’agitpasd’unservice;sonutilisationàdesfinsdetestoudesurveillanceestàéviter.\n\nДанныйдоменпредназначендляиспользованиявпримерахдокументациибезнеобходимостиполученияпредварительногоразрешения.Этонесервис;нерекомендуетсяегоиспользованиедлятестированияимониторинга.\n\nEstedominioestádestinadoalusoenejemplosdedocumentaciónsinnecesidaddepermiso.Estonoesunservicio,evitarutilizarlopararealizarpruebasomonitoreos.\n\n [Learn more](https://iana.org/help/example-domains)",
        "html": null,
        "raw_html": null,
        "summary": null
      },
      "media": {
        "screenshot_url": null,
        "audio_url": null,
        "video_url": null
      },
      "extraction": null,
      "answer": null,
      "highlights": null,
      "change_tracking": null,
      "page_count": null,
      "total_page_count": null,
      "fetch": {
        "cache_state": "hit",
        "cached_at": "2026-10-02T12:40:55.009Z",
        "proxy_tier": null
      },
      "page_state": {
        "value": "content",
        "confidence": 0.85,
        "skipped_judge": false
      }
    }
  },
  "credits_used": 1,
  "request_id": "req_example000000",
  "cached": false,
  "credits_remaining": 9999,
  "source": "captured",
  "captured_at": "2026-10-02T12:44:23.265Z",
  "redacted": true
}

통합 응답 구조를 보여 주는 실제 샘플이에요. 필드 값은 조회한 대상에 따라 달라져요.

API 상세

Web Scraping Scrape API는 어떻게 동작하나요

API 키와 함께 GET 요청을 보내면, 통합 스키마와 계산 필드를 담은 깔끔한 JSON이 돌아와요.

메서드

GET

응답 형식

JSON

왜 SocialCrawl인가요

Web Scraping Scrape 데이터는 왜 SocialCrawl로 가져올까요

Web Scraping 데이터 추출에서 까다로운 부분은 SocialCrawl이 가려 드려요. 통합 스키마, AI 인리치먼트, 그리고 코드에 남지 않는 플랫폼 로직까지.

개발자 우선

소셜 미디어 데이터를 몇 초 만에 수집하는 방법

개발자를 위한 가장 빠른 소셜 미디어 스크래핑 API. 월간 활성 사용자 100억 명 이상을 포괄하는 95개 플랫폼에서 프로필, 게시물, 댓글, 분석 데이터를 수집하세요.

모든 플랫폼을 하나의 스키마로

동일한 응답 구조로 95개 플랫폼을 조회하세요. 연동은 한 번이면 충분합니다.

단순 수집을 넘어 계산된 필드 제공

엔드포인트가 해당 지표를 지원하고 계산에 필요한 원본 값이 있을 때, 정규화된 레코드에 engagement_rate, estimated_reach, content_category, language를 함께 담아 바로 활용할 수 있습니다.

코드 한 줄 쓰기 전에, 데이터부터

Visual Data Explorer에 URL만 붙여넣으면 결과 카드와 정형화된 테이블, CSV 내보내기를 바로 사용할 수 있습니다.

import requests

response = requests.get(
    'https://www.socialcrawl.dev/v1/tiktok/profile',
    params={'handle': 'charlidamelio'},
    headers={'x-api-key': 'sc_YOUR_API_KEY'}
)
data = response.json()
[ .JSON ]
{
  "success": true,
  "platform": "tiktok",
  "data": {
    "author": {
      "username": "charlidamelio",
      "followers": 152400000
    },
    "engagement": {
      "likes": 12400000000,
      "engagement_rate": 0.087
    },
    "metadata": {
      "language": "en",
      "content_category": "lifestyle"
    }
  }
}
+ 95개 플랫폼
자주 묻는 질문

자주 묻는 질문

API, 요금제, 기능에 대한 질문과 답변입니다.

문의하기
Web Scrape API는 어떤 데이터를 돌려주나요?
웹 페이지 하나의 내용을 깔끔한 마크다운이나 HTML로 반환합니다. 최종 URL과 상태 코드, 수집 메타데이터가 함께 오고 원하면 스크린샷까지 붙습니다. 응답은 다른 SocialCrawl 엔드포인트와 같은 JSON 구조로 오고 data, credits_used, request_id 필드도 함께 담깁니다.
Web Scrape 엔드포인트는 언제 쓰면 좋나요?
사이트 전체가 대상이라면 crawl 작업을 걸어야 합니다. URL이 이미 정해진 한 페이지를 가져올 때 사용하세요.
Web Scrape API 비용은 얼마인가요?
요청 내용에 따라 호출당 1~5크레딧입니다. 가입하면 100크레딧을 무료로 드립니다. 실패했거나 빈 결과가 나온 호출의 크레딧은 자동으로 돌려드립니다. 구독료는 따로 없습니다.
Web Scrape API에는 어떤 파라미터가 필요한가요?
요청에는 url 파라미터가 꼭 들어가야 합니다. 선택 파라미터 14개는 이 페이지에 모두 정리해 두었습니다.
Web Scrape API는 어떻게 호출하나요?
https://www.socialcrawl.dev/v1/web/scrape?url=https%3A%2F%2Fexample.com 주소로 GET 요청을 보내세요. API 키는 x-api-key 헤더에 넣습니다. Authorization: Bearer 헤더도 쓸 수 있습니다.

AI에게 SocialCrawl을 물어보세요

Web Scraping Scrape 데이터, 가져올 준비 되셨어요?

API 키 받고 60초 안에 Web Scraping 데이터를 받아 가세요.

무료로 시작하기

🤖 AI 에이전트나 LLM이신가요? 이 페이지를 markdown으로 읽어보세요