Pagination

Every list endpoint on every platform paginates the same way. Learn it once and it works on all 193 paginatable endpoints: TikTok, Instagram, YouTube, Reddit, LinkedIn, Naver, all of them.

The one rule

Read pagination.next_cursor from the response, send it back as the cursor query parameter, and stop when pagination.has_more is false.

That is the entire contract. You never construct a cursor, decode a token, or look up a platform's native pagination param. The API hands you an opaque token and takes it straight back.

The response block

Every list response carries a top-level pagination block:

{
  "success": true,
  "platform": "tiktok",
  "endpoint": "/v1/tiktok/profile/videos",
  "data": {
    "items": [ /* … this page of results … */ ],
    "total": 1247
  },
  "credits_used": 1,
  "credits_remaining": 98,
  "request_id": "req-abc123",
  "cached": false,
  "pagination": {
    "next_cursor": "sc.eyJ2IjoyLCJjIjoiNzM4…",
    "has_more": true,
    "page_size": 30
  }
}

The token is a single sc. string that works identically for cursor-, page-, and offset-style upstreams, so your loop is byte-for-byte the same everywhere. (data.total, when present, is the upstream's own result count, useful for a progress bar, but never use it to decide when to stop; has_more is the only always-correct signal.)

A worked round-trip

Request page 1 (no cursor):

curl "https://www.socialcrawl.dev/v1/tiktok/profile/videos?handle=mrbeast" \
  -H "x-api-key: YOUR_API_KEY"

The response comes back with "pagination": { "next_cursor": "sc.eyJ2Ijoy…", "has_more": true, "page_size": 30 }.

Request page 2 (pass that next_cursor back as cursor):

curl "https://www.socialcrawl.dev/v1/tiktok/profile/videos?handle=mrbeast&cursor=sc.eyJ2Ijoy…" \
  -H "x-api-key: YOUR_API_KEY"

Keep going until a response returns "has_more": false (and "next_cursor": null). Swap tiktok/profile/videos for any list endpoint in the table below and this is unchanged.

The loop

Identical for every list endpoint on every platform:

async function* paginate(path: string, params: Record<string, string>) {
  let cursor: string | null = null;
  do {
    const url = new URL(`https://www.socialcrawl.dev/v1/${path}`);
    for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
    if (cursor) url.searchParams.set("cursor", cursor); // <-- always "cursor"

    const res = await fetch(url, {
      headers: { "x-api-key": process.env.SOCIALCRAWL_KEY! },
    });
    const json = await res.json();

    for (const item of json.data.items) yield item;

    cursor = json.pagination.next_cursor; // <-- send back verbatim next round
  } while (cursor); // <-- pagination.has_more === false ends the loop
}

// Usage: same call shape for TikTok, Instagram, YouTube, Reddit, …
for await (const video of paginate("tiktok/profile/videos", { handle: "mrbeast" })) {
  // … handle each item
}

Two things to get right:

1. Send next_cursor back as the EXACT string you received, under the key cursor. Don't decode, trim, or re-encode it; it's an opaque token.
2. Stop when has_more is false (equivalently, when next_cursor is null). Don't infer the end from an empty page or from total.

Recipe: drain a full account safely

The real-world loop is "pull every post/comment/follower an account has" without burning credits or tripping limits. Walk pages until has_more is false, and keep three things in mind:

// Drain every video for a handle, then stop.
async function drain(path: string, params: Record<string, string>) {
  const all: unknown[] = [];
  let cursor: string | null = null;
  let pages = 0;

  do {
    const url = new URL(`https://www.socialcrawl.dev/v1/${path}`);
    for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
    if (cursor) url.searchParams.set("cursor", cursor);

    const res = await fetch(url, {
      headers: { "x-api-key": process.env.SOCIALCRAWL_KEY! },
    });

    // Respect backpressure: on 429 (CONCURRENCY_LIMIT) or 503 back off and retry
    // the SAME cursor; never advance past a page you didn't successfully read.
    if (res.status === 429 || res.status === 503) {
      const wait = Number(res.headers.get("retry-after") ?? "2") * 1000;
      await new Promise((r) => setTimeout(r, wait));
      continue;
    }

    const json = await res.json();
    all.push(...json.data.items);
    cursor = json.pagination.next_cursor;
    pages += 1;
  } while (cursor);

  return { items: all, pages };
}

• Concurrency. You can drain many accounts in parallel, but a single API key allows at most 50 concurrent requests. Exceed it and you get a 429 CONCURRENCY_LIMIT (0 credits charged). Cap your worker pool well under 50; within one account, pages are inherently sequential (each needs the previous next_cursor).
• Credit budget. Each page is a normal billed request: one page of a 1-credit endpoint costs 1 credit, so draining a 50-page feed costs about 50 credits. Multiply pages by the endpoint's per-call cost (see Endpoint pricing) before you loop, and cap pages if you only need the most recent N results. Cache hits on re-runs cost 0 credits.
• Bounded depth. Page- and offset-style endpoints (Naver, LinkedIn, the app stores) have a finite maximum depth; has_more flips to false when you reach it. Cursor-style feeds end when the upstream runs out of items.

Sending the wrong param is a 400, not a silent bug

You never need a platform's native pagination param name: always send cursor. If you do send a foreign cursor param (say you copy max_id off an Instagram response onto a TikTok endpoint), you get a 400 INVALID_REQUEST with a did_you_mean: "cursor" hint and zero credits charged, instead of silently receiving page 1 again.

Stale or malformed cursors

If you pass a cursor the upstream rejects (expired, hand-edited, copied across endpoints), you'll get a 502 UPSTREAM_ERROR or 400 INVALID_REQUEST envelope. Credits are refunded on upstream errors (see Credits → When are credits refunded?). Start over with no cursor.

Two kinds of limit

limit means one of two different things depending on the endpoint, and the difference changes both how many requests you make and what you pay. The limit behaviour column in the tables below tells you which one you have.

Page size: caps a single upstream page. You still paginate yourself: one request returns one page, and you send cursor back for the next. The API maps limit to the upstream's native name (Naver display, GitHub per_page, and so on), capped at that endpoint's maximum. This is the common case.

Collect-until-N: the endpoint paginates for you. It walks upstream pages server-side until it has limit unique items (or the source runs dry), de-duplicates as it goes, and returns them in a single response. You are billed per page or window actually consumed, and the unused budget is refunded. This is how you get more than one window's worth of results in one call.

These endpoints collect until N:

An n/a in the column means no limit control at all: to get fewer items, slice client-side; to get more, paginate another page.

Two things that are not limit: /v1/prism/comments takes both max and limit: max bounds how many pages it scans (and is what you pay for), while limit only caps how many of the scanned comments come back, so it never changes the price. And pagination.page_size in the response is a report of what arrived, not a control.

Every paginatable endpoint

193 endpoints paginate today (113 cursor-based, 62 page-based, 18 offset-based). Every one of them takes the universal cursor input and returns the same pagination block (next_cursor / has_more / page_size): no per-platform special cases. The Style column is informational only; you always send cursor.

AliExpress

Amazon

Apple App Store

Content Analysis

eBay

Etsy

Facebook

G2

GitHub

Google

Google Play

Gumtree

H&M

Hacker News

Home Depot

Instagram

Jobs

Klarna

Kohl's

Kwai

LinkedIn

Naver

Pinterest

Prism

Reddit

Rumble

Sephora

Snapchat

Spotify

Target

Telegram

Threads

TikTok

TikTok Shop

Truth Social

Twitter/X

US Congress Trades

Walmart

Wayfair

Web Scraping

Yelp

YouTube

See also

• Response schema → List endpoints: the envelope shape in one sentence.
• Quickstart: your first request in under a minute.
• Endpoint pricing: every endpoint and what each call costs.
• Error handling: refund rules for stale-cursor errors, and the concurrency limit.