100 크레딧 무료, 카드 등록 없이지금 시작하기
Logo
Web Scraping logo
100 크레딧 무료, 카드 등록 없이

Web Scraping Map API

사이트에 있는 URL 목록을 각 페이지 내용은 가져오지 않고 반환합니다.

GET /v1/web/map

2026년 11월 업데이트SocialCrawl 팀이 직접 관리해요

Web Scraping Map API, 직접 써 보세요

코드 한 줄 짜기 전에, 실제 데이터부터

GET/v1/web/map

Site URL to map.

선택 파라미터 7개

Optional path or keyword filter.

Maximum URLs to return, up to 5000.

How to use the site's sitemap.xml: 'include' (default) to merge it with crawled links, 'only' to trust it alone, 'skip' to ignore it.

Follow links onto subdomains of the root URL. Default true.

Treat URLs differing only by query string as one URL, which keeps paginated and tracking-tagged duplicates out of the result. Default true.

Bypass the upstream cache and re-discover the site now. Slower; use it when the map is known to be stale.

Optional. What you are looking for on the site, in your own words. The map still returns every URL, ranked, with skipped_urls listing the ones that look unlikely to hold it (each with p). Without this param the map is unchanged.

Web Scraping Map API는 언제 쓰나요

어느 페이지를 scrape하거나 crawl할지 고르기 전에 사이트 구조부터 볼 때 사용하세요.

API 레퍼런스 영문 원문

Discovers URLs for a public site and returns them as a normalized WebPageList.

95개 플랫폼을 하나의 API로 제공합니다

·TikTok logoTikTok·Instagram logoInstagram·YouTube logoYouTube·Facebook logoFacebook·X logoX·LinkedIn logoLinkedIn·Reddit logoReddit·Threads logoThreads·Pinterest logoPinterest·Twitch logoTwitch·Truth Social logoTruth Social·Snapchat logoSnapchat·Kick logoKick·Bluesky logoBluesky·Kwai logoKwai·Rumble logoRumble·Spotify logoSpotify·Apple Music logoApple Music·TikTok Shop logoTikTok Shop·Amazon Shop logoAmazon Shop·Google Shopping logoGoogle Shopping·Trustpilot logoTrustpilot·TripAdvisor logoTripAdvisor·Yelp logoYelp·Linktree logoLinktree·Komi logoKomi·Pillar logoPillar·lnk.bio logolnk.bio·Facebook Ads logoFacebook Ads·Google Ads logoGoogle Ads·LinkedIn Ads logoLinkedIn Ads·Google Search logoGoogle Search·Google News logoGoogle News·Finance logoFinance·Polymarket logoPolymarket·Tavily logoTavily·Hacker News logoHacker News·GitHub logoGitHub·Perplexity logoPerplexity·Naver logoNaver·Utility logoUtility·Universal Search logoUniversal Search
·TikTok logoTikTok·Instagram logoInstagram·YouTube logoYouTube·Facebook logoFacebook·X logoX·LinkedIn logoLinkedIn·Reddit logoReddit·Threads logoThreads·Pinterest logoPinterest·Twitch logoTwitch·Truth Social logoTruth Social·Snapchat logoSnapchat·Kick logoKick·Bluesky logoBluesky·Kwai logoKwai·Rumble logoRumble·Spotify logoSpotify·Apple Music logoApple Music·TikTok Shop logoTikTok Shop·Amazon Shop logoAmazon Shop·Google Shopping logoGoogle Shopping·Trustpilot logoTrustpilot·TripAdvisor logoTripAdvisor·Yelp logoYelp·Linktree logoLinktree·Komi logoKomi·Pillar logoPillar·lnk.bio logolnk.bio·Facebook Ads logoFacebook Ads·Google Ads logoGoogle Ads·LinkedIn Ads logoLinkedIn Ads·Google Search logoGoogle Search·Google News logoGoogle News·Finance logoFinance·Polymarket logoPolymarket·Tavily logoTavily·Hacker News logoHacker News·GitHub logoGitHub·Perplexity logoPerplexity·Naver logoNaver·Utility logoUtility·Universal Search logoUniversal Search
·TikTok logoTikTok·Instagram logoInstagram·YouTube logoYouTube·Facebook logoFacebook·X logoX·LinkedIn logoLinkedIn·Reddit logoReddit·Threads logoThreads·Pinterest logoPinterest·Twitch logoTwitch·Truth Social logoTruth Social·Snapchat logoSnapchat·Kick logoKick·Bluesky logoBluesky·Kwai logoKwai·Rumble logoRumble·Spotify logoSpotify·Apple Music logoApple Music·TikTok Shop logoTikTok Shop·Amazon Shop logoAmazon Shop·Google Shopping logoGoogle Shopping·Trustpilot logoTrustpilot·TripAdvisor logoTripAdvisor·Yelp logoYelp·Linktree logoLinktree·Komi logoKomi·Pillar logoPillar·lnk.bio logolnk.bio·Facebook Ads logoFacebook Ads·Google Ads logoGoogle Ads·LinkedIn Ads logoLinkedIn Ads·Google Search logoGoogle Search·Google News logoGoogle News·Finance logoFinance·Polymarket logoPolymarket·Tavily logoTavily·Hacker News logoHacker News·GitHub logoGitHub·Perplexity logoPerplexity·Naver logoNaver·Utility logoUtility·Universal Search logoUniversal Search
·TikTok logoTikTok·Instagram logoInstagram·YouTube logoYouTube·Facebook logoFacebook·X logoX·LinkedIn logoLinkedIn·Reddit logoReddit·Threads logoThreads·Pinterest logoPinterest·Twitch logoTwitch·Truth Social logoTruth Social·Snapchat logoSnapchat·Kick logoKick·Bluesky logoBluesky·Kwai logoKwai·Rumble logoRumble·Spotify logoSpotify·Apple Music logoApple Music·TikTok Shop logoTikTok Shop·Amazon Shop logoAmazon Shop·Google Shopping logoGoogle Shopping·Trustpilot logoTrustpilot·TripAdvisor logoTripAdvisor·Yelp logoYelp·Linktree logoLinktree·Komi logoKomi·Pillar logoPillar·lnk.bio logolnk.bio·Facebook Ads logoFacebook Ads·Google Ads logoGoogle Ads·LinkedIn Ads logoLinkedIn Ads·Google Search logoGoogle Search·Google News logoGoogle News·Finance logoFinance·Polymarket logoPolymarket·Tavily logoTavily·Hacker News logoHacker News·GitHub logoGitHub·Perplexity logoPerplexity·Naver logoNaver·Utility logoUtility·Universal Search logoUniversal Search
Web Scraping API

Map API로 무엇을 할 수 있을까요

Map 엔드포인트가 통합 스키마와 계산 필드를 담은 Web Scraping 데이터를 한 번의 요청으로 보내드려요. 스크래핑 인프라를 직접 만들거나 유지할 필요가 없어요.

요청 예시

# GET /v1/web/map: 1 credit per call. Cache hits are free and failed calls are refunded.
# Check the price first for 0 credits: add dry_run=1 to the query. Nothing is fetched or charged.
# --max-time 34: this endpoint's measured p99 latency x 1.5.
# --retry 3: retries timeouts, 429 and 5xx, honoring Retry-After.
# -o response.json keeps only the final attempt's body.
# One Idempotency-Key per request, reused on every retry, so a retried call is never charged twice.
IDEMPOTENCY_KEY=$(od -An -N16 -tx1 /dev/urandom | tr -d ' \n')
curl -sS --max-time 34 --retry 3 -o response.json \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Idempotency-Key: $IDEMPOTENCY_KEY" \
  "https://www.socialcrawl.dev/v1/web/map?url=https%3A%2F%2Fexample.com&limit=100" && cat response.json
# GET /v1/web/map: 1 credit per call. Cache hits are free and failed calls are refunded.
# Check the price first for 0 credits: add dry_run=1 to the query. Nothing is fetched or charged.
import os
import random
import time
import uuid

import requests

API_KEY = os.environ.get("SOCIALCRAWL_API_KEY", "YOUR_API_KEY")
TIMEOUT_S = 34  # this endpoint's measured p99 latency x 1.5.
MAX_RETRIES = 3
URL = "https://www.socialcrawl.dev/v1/web/map"


def backoff(attempt):
    # Exponential backoff with jitter: about 1 s, 2 s, then 4 s.
    return 2 ** attempt * (0.5 + random.random() / 2)


def socialcrawl(params):
    # One Idempotency-Key per request, reused on every retry, so a retried call is never charged twice.
    idempotency_key = str(uuid.uuid4())
    for attempt in range(MAX_RETRIES + 1):
        try:
            res = requests.get(
                URL,
                params=params,
                headers={"x-api-key": API_KEY, "Idempotency-Key": idempotency_key},
                timeout=TIMEOUT_S,
            )
            body = res.json()
        except (requests.RequestException, ValueError):
            # A timeout, a dropped connection or a non-JSON gateway page: retry with the same key.
            if attempt == MAX_RETRIES:
                raise
            time.sleep(backoff(attempt))
            continue
        if body["success"]:
            return body
        error = body["error"]
        # Branch on error["type"], never on the message text.
        if error["type"] == "RESOURCE_NOT_FOUND":
            return None  # refunded: nothing to read
        retry_after = res.headers.get("Retry-After")
        # SERVICE_UNAVAILABLE without Retry-After is a withdrawn endpoint: permanent.
        permanent = error["type"] == "SERVICE_UNAVAILABLE" and retry_after is None
        if not error.get("retryable") or permanent or attempt == MAX_RETRIES:
            raise RuntimeError(f"{error['type']}: {error['message']} (request_id {body['request_id']})")
        # Retry-After (seconds) wins; otherwise back off.
        time.sleep(float(retry_after) if retry_after else backoff(attempt))


params = {
    "url": "https://example.com",
    "limit": "100",
}
body = socialcrawl(params)
if body is not None:
    print(body["data"])
// GET /v1/web/map: 1 credit per call. Cache hits are free and failed calls are refunded.
// Check the price first for 0 credits: add dry_run=1 to the query. Nothing is fetched or charged.
const API_KEY = process.env.SOCIALCRAWL_API_KEY ?? "YOUR_API_KEY";
const TIMEOUT_MS = 34_000; // this endpoint's measured p99 latency x 1.5.
const MAX_RETRIES = 3;

type Page = { next_cursor: string | null; has_more: boolean };
type SocialCrawlResponse =
  | { success: true; data: any; pagination?: Page; credits_used: number; request_id: string }
  | { success: false; error: { type: string; message: string; retryable: boolean }; request_id: string };

const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
// Exponential backoff with jitter: about 1 s, 2 s, then 4 s.
const backoffMs = (attempt: number) => 2 ** attempt * 1000 * (0.5 + Math.random() / 2);

async function socialcrawl(url: URL) {
  // One Idempotency-Key per request, reused on every retry, so a retried call is never charged twice.
  const idempotencyKey = crypto.randomUUID();
  for (let attempt = 0; ; attempt++) {
    let res: Response;
    let body: SocialCrawlResponse;
    try {
      res = await fetch(url, {
        headers: {
          "x-api-key": API_KEY,
          "Idempotency-Key": idempotencyKey,
        },
        signal: AbortSignal.timeout(TIMEOUT_MS),
      });
      body = (await res.json()) as SocialCrawlResponse;
    } catch (error) {
      // A timeout, a dropped connection or a non-JSON gateway page: retry with the same key.
      if (attempt >= MAX_RETRIES) throw error;
      await sleep(backoffMs(attempt));
      continue;
    }
    if (body.success) return body;
    // Branch on error.type, never on the message text.
    if (body.error.type === "RESOURCE_NOT_FOUND") return null; // refunded: nothing to read
    const retryAfter = res.headers.get("Retry-After");
    // SERVICE_UNAVAILABLE without Retry-After is a withdrawn endpoint: permanent.
    const permanent = body.error.type === "SERVICE_UNAVAILABLE" && retryAfter === null;
    if (!body.error.retryable || permanent || attempt >= MAX_RETRIES) {
      throw new Error(`${body.error.type}: ${body.error.message} (request_id ${body.request_id})`);
    }
    // Retry-After (seconds) wins; otherwise back off.
    await sleep(retryAfter ? Number(retryAfter) * 1000 : backoffMs(attempt));
  }
}

const url = new URL("https://www.socialcrawl.dev/v1/web/map");
url.searchParams.set("url", "https://example.com");
url.searchParams.set("limit", "100");

const body = await socialcrawl(url);
if (body) console.log(body.data);

파라미터

파라미터필수설명
url예Site URL to map.
search아니오Optional path or keyword filter.
limit아니오Maximum URLs to return, up to 5000.
sitemap아니오How to use the site's sitemap.xml: 'include' (default) to merge it with crawled links, 'only' to trust it alone, 'skip' to ignore it. (include | skip | only)
include_subdomains아니오Follow links onto subdomains of the root URL. Default true.
ignore_query_parameters아니오Treat URLs differing only by query string as one URL, which keeps paginated and tracking-tagged duplicates out of the result. Default true.
fresh아니오Bypass the upstream cache and re-discover the site now. Slower; use it when the map is known to be stale.
goal아니오Optional. What you are looking for on the site, in your own words. The map still returns every URL, ranked, with skipped_urls listing the ones that look unlikely to hold it (each with p). Without this param the map is unchanged.
응답 예시

Web Scraping Map API는 무엇을 돌려주나요

모든 응답은 하나의 통합 스키마를 따라요. 크레딧을 쓰기 전에 어떤 필드가 돌아오는지, 실제 응답 본문 그대로 확인해 보세요.

응답 예시 보기
{
  "success": true,
  "platform": "web",
  "endpoint": "/v1/web/map",
  "data": {
    "items": [
      {
        "page": {
          "url": "https://example.com/",
          "final_url": "https://example.com/",
          "status_code": null,
          "scrape_id": null,
          "fetched_at": null,
          "title": "Example Domain",
          "description": null,
          "content": {
            "markdown": null,
            "html": null,
            "raw_html": null,
            "summary": null
          },
          "media": {
            "screenshot_url": null,
            "audio_url": null,
            "video_url": null
          },
          "extraction": null,
          "answer": null,
          "highlights": null,
          "change_tracking": null,
          "page_count": null,
          "total_page_count": null,
          "fetch": {
            "cache_state": null,
            "cached_at": null,
            "proxy_tier": null
          }
        }
      },
      {
        "page": {
          "url": "https://www.example.com/",
          "final_url": "https://www.example.com/",
          "status_code": null,
          "scrape_id": null,
          "fetched_at": null,
          "title": "Example Domain",
          "description": null,
          "content": {
            "markdown": null,
            "html": null,
            "raw_html": null,
            "summary": null
          },
          "media": {
            "screenshot_url": null,
            "audio_url": null,
            "video_url": null
          },
          "extraction": null,
          "answer": null,
          "highlights": null,
          "change_tracking": null,
          "page_count": null,
          "total_page_count": null,
          "fetch": {
            "cache_state": null,
            "cached_at": null,
            "proxy_tier": null
          }
        }
      }
    ],
    "total": 3,
    "dropped": 0
  },
  "credits_used": 1,
  "request_id": "req_example000000",
  "cached": false,
  "pagination": {
    "next_cursor": null,
    "has_more": false,
    "page_size": 3
  },
  "credits_remaining": 9999,
  "source": "captured",
  "captured_at": "2026-10-02T12:44:25.365Z",
  "redacted": true
}

통합 응답 구조를 보여 주는 실제 샘플이에요. 필드 값은 조회한 대상에 따라 달라져요.

API 상세

Web Scraping Map API는 어떻게 동작하나요

API 키와 함께 GET 요청을 보내면, 통합 스키마와 계산 필드를 담은 깔끔한 JSON이 돌아와요.

메서드

GET

응답 형식

JSON

왜 SocialCrawl인가요

Web Scraping Map 데이터는 왜 SocialCrawl로 가져올까요

Web Scraping 데이터 추출에서 까다로운 부분은 SocialCrawl이 가려 드려요. 통합 스키마, AI 인리치먼트, 그리고 코드에 남지 않는 플랫폼 로직까지.

개발자 우선

소셜 미디어 데이터를 몇 초 만에 수집하는 방법

개발자를 위한 가장 빠른 소셜 미디어 스크래핑 API. 월간 활성 사용자 100억 명 이상을 포괄하는 95개 플랫폼에서 프로필, 게시물, 댓글, 분석 데이터를 수집하세요.

모든 플랫폼을 하나의 스키마로

동일한 응답 구조로 95개 플랫폼을 조회하세요. 연동은 한 번이면 충분합니다.

단순 수집을 넘어 계산된 필드 제공

엔드포인트가 해당 지표를 지원하고 계산에 필요한 원본 값이 있을 때, 정규화된 레코드에 engagement_rate, estimated_reach, content_category, language를 함께 담아 바로 활용할 수 있습니다.

코드 한 줄 쓰기 전에, 데이터부터

Visual Data Explorer에 URL만 붙여넣으면 결과 카드와 정형화된 테이블, CSV 내보내기를 바로 사용할 수 있습니다.

import requests

response = requests.get(
    'https://www.socialcrawl.dev/v1/tiktok/profile',
    params={'handle': 'charlidamelio'},
    headers={'x-api-key': 'sc_YOUR_API_KEY'}
)
data = response.json()
[ .JSON ]
{
  "success": true,
  "platform": "tiktok",
  "data": {
    "author": {
      "username": "charlidamelio",
      "followers": 152400000
    },
    "engagement": {
      "likes": 12400000000,
      "engagement_rate": 0.087
    },
    "metadata": {
      "language": "en",
      "content_category": "lifestyle"
    }
  }
}
+ 95개 플랫폼
자주 묻는 질문

자주 묻는 질문

API, 요금제, 기능에 대한 질문과 답변입니다.

문의하기
Web Map API는 어떤 데이터를 돌려주나요?
사이트에 있는 URL 목록을 각 페이지 내용은 가져오지 않고 반환합니다. 응답은 다른 SocialCrawl 엔드포인트와 같은 JSON 구조로 오고 data, credits_used, request_id 필드도 함께 담깁니다.
Web Map 엔드포인트는 언제 쓰면 좋나요?
어느 페이지를 scrape하거나 crawl할지 고르기 전에 사이트 구조부터 볼 때 사용하세요.
Web Map API 비용은 얼마인가요?
호출당 1크레딧입니다. 가입하면 받는 무료 100크레딧으로 100번 호출할 수 있습니다. 실패했거나 빈 결과가 나온 호출의 크레딧은 자동으로 돌려드립니다. 구독료는 따로 없습니다.
Web Map API에는 어떤 파라미터가 필요한가요?
요청에는 url 파라미터가 꼭 들어가야 합니다. 선택 파라미터 7개는 이 페이지에 모두 정리해 두었습니다.
Web Map API는 어떻게 호출하나요?
https://www.socialcrawl.dev/v1/web/map?url=https%3A%2F%2Fexample.com 주소로 GET 요청을 보내세요. API 키는 x-api-key 헤더에 넣습니다. Authorization: Bearer 헤더도 쓸 수 있습니다.

AI에게 SocialCrawl을 물어보세요

Web Scraping Map 데이터, 가져올 준비 되셨어요?

API 키 받고 60초 안에 Web Scraping 데이터를 받아 가세요.

무료로 시작하기

🤖 AI 에이전트나 LLM이신가요? 이 페이지를 markdown으로 읽어보세요