Pagination

Every list endpoint on every platform paginates the same way. Learn it once and it works on all 216 paginatable endpoints: TikTok, Instagram, YouTube, Reddit, LinkedIn, Naver, all of them.

The one rule

Read pagination.next_cursor from the response, send it back as the cursor query parameter, and stop when pagination.has_more is false.

That is the entire contract. You never construct a cursor, decode a token, or look up a platform's native pagination param. The API hands you an opaque token and takes it straight back.

The response block

Every list response carries a top-level pagination block:

{
  "success": true,
  "platform": "tiktok",
  "endpoint": "/v1/tiktok/profile/videos",
  "data": {
    "items": [ /* … this page of results … */ ],
    "total": 1247
  },
  "credits_used": 1,
  "credits_remaining": 98,
  "request_id": "req-abc123",
  "cached": false,
  "pagination": {
    "next_cursor": "sc.eyJ2IjoyLCJjIjoiNzM4…",
    "has_more": true,
    "page_size": 30
  }
}

The token is a single sc. string that works identically for cursor-, page-, and offset-style upstreams, so your loop is byte-for-byte the same everywhere. (data.total, when present, is the upstream's own result count, useful for a progress bar, but never use it to decide when to stop; has_more is the only always-correct signal.)

A worked round-trip

Request page 1 (no cursor):

curl "https://www.socialcrawl.dev/v1/tiktok/profile/videos?handle=mrbeast" \
  -H "x-api-key: YOUR_API_KEY"

The response comes back with "pagination": { "next_cursor": "sc.eyJ2Ijoy…", "has_more": true, "page_size": 30 }.

Request page 2 (pass that next_cursor back as cursor):

curl "https://www.socialcrawl.dev/v1/tiktok/profile/videos?handle=mrbeast&cursor=sc.eyJ2Ijoy…" \
  -H "x-api-key: YOUR_API_KEY"

Keep going until a response returns "has_more": false (and "next_cursor": null). Swap tiktok/profile/videos for any list endpoint in the table below and this is unchanged.

The loop

Identical for every list endpoint on every platform:

async function* paginate(path: string, params: Record<string, string>) {
  let cursor: string | null = null;
  do {
    const url = new URL(`https://www.socialcrawl.dev/v1/${path}`);
    for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
    if (cursor) url.searchParams.set("cursor", cursor); // <-- always "cursor"

    const res = await fetch(url, {
      headers: { "x-api-key": process.env.SOCIALCRAWL_KEY! },
    });
    const json = await res.json();

    for (const item of json.data.items) yield item;

    cursor = json.pagination.next_cursor; // <-- send back verbatim next round
  } while (cursor); // <-- pagination.has_more === false ends the loop
}

// Usage: same call shape for TikTok, Instagram, YouTube, Reddit, …
for await (const video of paginate("tiktok/profile/videos", { handle: "mrbeast" })) {
  // … handle each item
}

Two things to get right:

1. Send next_cursor back as the EXACT string you received, under the key cursor. Don't decode, trim, or re-encode it; it's an opaque token.
2. Stop when has_more is false (equivalently, when next_cursor is null). Don't infer the end from an empty page or from total.

Recipe: drain a full account safely

The real-world loop is "pull every post/comment/follower an account has" without burning credits or tripping limits. Walk pages until has_more is false, and keep three things in mind:

// Drain every video for a handle, then stop.
async function drain(path: string, params: Record<string, string>) {
  const all: unknown[] = [];
  let cursor: string | null = null;
  let pages = 0;

  do {
    const url = new URL(`https://www.socialcrawl.dev/v1/${path}`);
    for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
    if (cursor) url.searchParams.set("cursor", cursor);

    const res = await fetch(url, {
      headers: { "x-api-key": process.env.SOCIALCRAWL_KEY! },
    });

    // Respect backpressure: on 429 (CONCURRENCY_LIMIT) or 503 back off and retry
    // the SAME cursor; never advance past a page you didn't successfully read.
    if (res.status === 429 || res.status === 503) {
      const wait = Number(res.headers.get("retry-after") ?? "2") * 1000;
      await new Promise((r) => setTimeout(r, wait));
      continue;
    }

    const json = await res.json();
    all.push(...json.data.items);
    cursor = json.pagination.next_cursor;
    pages += 1;
  } while (cursor);

  return { items: all, pages };
}

• Concurrency. You can drain many accounts in parallel, but a single API key allows at most 50 concurrent requests. Exceed it and you get a 429 CONCURRENCY_LIMIT (0 credits charged). Cap your worker pool well under 50; within one account, pages are inherently sequential (each needs the previous next_cursor).
• Credit budget. Each page is a normal billed request: one page of a 1-credit endpoint costs 1 credit, so draining a 50-page feed costs about 50 credits. Multiply pages by the endpoint's per-call cost (see Endpoint pricing) before you loop, and cap pages if you only need the most recent N results. Cache hits on re-runs cost 0 credits.
• Bounded depth. Page- and offset-style endpoints (Naver, LinkedIn, the app stores) have a finite maximum depth; has_more flips to false when you reach it. Cursor-style feeds end when the upstream runs out of items.

Incremental sync

A poller that checks a creator every day only needs the posts published since its last run. Eight creator post lists take two parameters that end the walk where your last poll stopped, so you pay for the pages with new posts and no more:

curl "https://www.socialcrawl.dev/v1/instagram/profile/posts?handle=nasa&stop_at_id=3990495229280545049" \
  -H "x-api-key: YOUR_API_KEY"

When you send either parameter, the pagination block carries one more field:

"pagination": {
  "next_cursor": null,
  "has_more": false,
  "page_size": 5,
  "stopped_at": "known_id"
}

stopped_at is absent when you sent neither parameter, and a call without them is unchanged. The one rule still holds: stop when has_more is false.

A daily poll

Send the same stop_at_id (or since) on every page of the walk, then keep the newest post you received for the next run:

// Fetch only the posts published since the last poll.
async function pollNewPosts(
  path: string,
  params: Record<string, string>,
  newestHeldId: string | null,
) {
  const fresh: any[] = [];
  let cursor: string | null = null;
  do {
    const url = new URL(`https://www.socialcrawl.dev/v1/${path}`);
    for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
    // First run: bound the walk by date. Every later run: by the newest post held.
    if (newestHeldId) url.searchParams.set("stop_at_id", newestHeldId);
    else url.searchParams.set("since", "2026-09-01");
    if (cursor) url.searchParams.set("cursor", cursor);

    const res = await fetch(url, {
      headers: { "x-api-key": process.env.SOCIALCRAWL_KEY! },
    });
    const json = await res.json();
    fresh.push(...json.data.items);
    cursor = json.pagination.next_cursor; // null once the walk has stopped
  } while (cursor);

  // Pinned posts sit out of date order, so never keep one as the newest.
  const newest = fresh.find((row) => row.post.flags?.pinned !== true);
  return { fresh, newestHeldId: newest?.post.id ?? newestHeldId };
}

// savedId is the newestHeldId you stored after the previous run, or null.
const result = await pollNewPosts("tiktok/profile/videos", { handle: "mrbeast" }, savedId);

Three details decide whether a poll is correct and cheap:

• Pinned posts never end the walk. A profile opens with the posts its owner pinned, and those can be years old. A pinned post never counts as the boundary. If it is older than since, or it is the post you named in stop_at_id, it is left off the page and the walk goes on.
• The price does not change. Each page costs what a page of that endpoint costs, 1 credit on all eight. A daily poll that finds your newest post on page 1 pays for page 1 and stops there. A since that is not a date, or a stop_at_id longer than 512 characters, is a 400 INVALID_REQUEST and costs nothing.
• YouTube needs newest first. A stop only means something on a list in date order, so either parameter sets sort=latest on channel/videos and sort=newest on channel/shorts when you leave sort out, and sort=popular with either is a 400. YouTube list dates are estimated from labels such as "4 months ago", so use stop_at_id there for an exact stop.

Date window on every list

Every list whose rows carry a date takes since and until: posts, comments, replies, reviews and news articles, on every platform. The window is applied to each page before it is returned, so every dated row you receive is inside it.

curl "https://www.socialcrawl.dev/v1/youtube/video/comments?url=https://www.youtube.com/watch?v=dQw4w9WgXcQ&since=2025-10-07" \
  -H "x-api-key: YOUR_API_KEY"

How far a walk goes depends on whether the list is in date order:

• Lists in date order stop at since. On these lists since sets the newest-first sort when you leave it out, and the page that reaches an older row ends the walk with pagination.stopped_at set to since, exactly like Incremental sync. Another sort value with since is a 400. They are the creator feeds above and app_store/app-reviews (sort_by=most_recent), google_play/app-reviews (sort_by=newest), instagram/post/comments (sort=recent), linkedin/company/posts (sort_by=recent), linkedin/post/comments (sort_order=recent), naver/news/search (sort=date), reddit/profile/posts (sort=new), reddit/search (sort=new), reddit/search/comments (sort=new), reddit/subreddit (sort=new), tiktok/post/comments (sort=recent), tripadvisor/reviews (sort_by=most_recent), trustpilot/reviews (sort=recency), twitter/search/tweets (sort=latest), youtube/video/comments (order=newest).
• Every other list filters each page and keeps its cursor. Comments in relevance order, keyword search and most reviews are not in date order, so an old row on page 1 says nothing about page 2. Keep following next_cursor; a page can come back with no rows and a cursor when every row on it was outside the window.
• The price does not change. Each page costs what a page of that endpoint costs, whether or not rows were left off. A row with no date is kept. A since or until that is not a date, or a since later than until, is a 400 INVALID_REQUEST and costs nothing.
• Other spellings are read as the window. published_after, start_date, from_date and date_from are applied as since, and published_before, end_date, to_date and date_to as until, on any list that does not take them under its own name. A named window (publish_time or date_posted set to yesterday, this-week, this-month, last-3-months or last-6-months) and a day count (max_age_days=730) become a since that far back. A list that has its own date filter keeps it: youtube/search/advanced still takes published_after, and Threads search still takes start_date.
• A list whose rows carry no date refuses a date. Profile lists, product lists and suggestions return 400 INVALID_REQUEST with error.details.reason set to unsupported_date_filter, before any credit is charged, so a date key is never silently ignored.

Sending the wrong param is a 400, not a silent bug

You never need a platform's native pagination param name: always send cursor. If you do send a foreign cursor param (say you copy max_id off an Instagram response onto a TikTok endpoint), you get a 400 INVALID_REQUEST with a did_you_mean: "cursor" hint and zero credits charged, instead of silently receiving page 1 again.

Stale or malformed cursors

If you pass a cursor the upstream rejects (expired, hand-edited, copied across endpoints), you'll get a 502 UPSTREAM_ERROR or 400 INVALID_REQUEST envelope. Credits are refunded on upstream errors (see Credits → When are credits refunded?). Start over with no cursor.

Three kinds of limit

limit means one of three different things depending on the endpoint, and the difference changes both how many requests you make and what you pay. The limit behaviour column in the tables below tells you which one you have.

Page size: caps a single upstream page. You still paginate yourself: one request returns one page, and you send cursor back for the next. The API maps limit to the upstream's native name (Naver display, GitHub per_page, and so on), capped at that endpoint's maximum. This is the common case.

Collect-until-N: the endpoint paginates for you. It walks upstream pages server-side until it has limit unique items (or the source runs dry), de-duplicates as it goes, and returns them in a single response. You are billed per page or window actually consumed, and the unused budget is refunded. This is how you get more than one window's worth of results in one call.

These endpoints collect until N:

Row cap: on a lane that can fill its own rows with include, limit takes the top N rows of the page the search returned and caps the extra credits the join can spend with them, so limit=5&include=… holds the page price plus 5. It is applied after the page arrives and does not change the page the upstream served, so the cursor still advances past the whole page: rows beyond N are not returned by the next page either. Use it to buy a cheaper join, not to walk a list in smaller steps.

An n/a in the column means no limit control at all: to get fewer items, slice client-side; to get more, paginate another page.

Two things that are not limit: /v1/prism/comments takes both max and limit: max bounds how many pages it scans (and is what you pay for), while limit only caps how many of the scanned comments come back, so it never changes the price. And pagination.page_size in the response is a report of what arrived, not a control.

Every paginatable endpoint

216 endpoints paginate today (128 cursor-based, 70 page-based, 18 offset-based). Every one of them takes the universal cursor input and returns the same pagination block (next_cursor / has_more / page_size): no per-platform special cases. The Style column is informational only; you always send cursor.

AliExpress

Amazon

Apple App Store

Content Analysis

Douyin

eBay

Economy

Etsy

Facebook

G2

GitHub

Google

Google Play

Gumtree

H&M

Hacker News

Home Depot

Instagram

Jobs

Klarna

Kohl's

Kwai

LinkedIn

Naver

Pinterest

Prism

Reddit

Rumble

Sephora

Snapchat

Spotify

Target

Telegram

Threads

TikTok

TikTok Shop

Truth Social

Twitter/X

US Congress Trades

Walmart

Wayfair

Web Scraping

Xiaohongshu

Yelp

YouTube

See also

• Response schema → List endpoints: the envelope shape in one sentence.
• Quickstart: your first request in under a minute.
• Endpoint pricing: every endpoint and what each call costs.
• Error handling: refund rules for stale-cursor errors, and the concurrency limit.