Batch endpoints

Six endpoints take a whole array in one POST and return one row per input item. Use them when you need the same lookup for a list: a morning refresh of 400 post URLs, 60 handles to vet, 100 transcripts for a corpus.

Batch is not a different product. Each row is byte-identical to what the matching single endpoint returns. You trade N requests for one, and you pay for the rows that actually resolved.

The six endpoints

Authentication is the same x-api-key header as every other endpoint. These responses are never cached, so cached is always false. A batch refresh exists to read the live number.

However many items it carries, a batch counts as one request against the 600 per minute and 50 concurrent budgets. See Rate limits.

Making a request

Send a JSON body. The request body ceiling is 1 MB. A larger body is rejected with 413 PAYLOAD_TOO_LARGE before anything is parsed or charged.

curl -X POST 'https://www.socialcrawl.dev/v1/prism/profiles' \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: vetting-batch-1" \
  -d '{
    "items": [
      { "platform": "tiktok", "handle": "@scout2015" },
      { "platform": "linkedin", "handle": "williamhgates", "custom_id": "vet-1" },
      { "platform": "instagram", "handle": "@nosuchaccount" }
    ]
  }'

const BASE = "https://www.socialcrawl.dev";

const res = await fetch(`${BASE}/v1/prism/profiles`, {
  method: "POST",
  headers: {
    "x-api-key": process.env.SOCIALCRAWL_API_KEY!,
    "Content-Type": "application/json",
    "Idempotency-Key": "vetting-batch-1",
  },
  body: JSON.stringify({
    items: [
      { platform: "tiktok", handle: "@scout2015" },
      { platform: "linkedin", handle: "williamhgates", custom_id: "vet-1" },
      { platform: "instagram", handle: "@nosuchaccount" },
    ],
  }),
});

const { data } = await res.json();
console.log(data.summary.ok, data.summary.credits_charged);

import os

import requests

res = requests.post(
    "https://www.socialcrawl.dev/v1/prism/profiles",
    headers={
        "x-api-key": os.environ["SOCIALCRAWL_API_KEY"],
        "Idempotency-Key": "vetting-batch-1",
    },
    json={
        "items": [
            {"platform": "tiktok", "handle": "@scout2015"},
            {"platform": "linkedin", "handle": "williamhgates", "custom_id": "vet-1"},
            {"platform": "instagram", "handle": "@nosuchaccount"},
        ]
    },
)

summary = res.json()["data"]["summary"]
print(summary["ok"], summary["credits_charged"])

The response

The four fan-out endpoints (post-stats, comment-lookup, profiles, transcripts) return per-row results plus a summary:

{
  "success": true,
  "platform": "prism",
  "endpoint": "/v1/prism/profiles",
  "data": {
    "results": [
      {
        "index": 0,
        "target": { "platform": "tiktok", "handle": "@scout2015" },
        "platform": "tiktok",
        "status": "ok",
        "data": { "author": { "username": "scout2015", "followers": 5300 } },
        "cost": 1,
        "fetched_at": "2026-07-04T09:00:03.128Z"
      },
      {
        "index": 1,
        "custom_id": "vet-1",
        "target": { "platform": "linkedin", "handle": "williamhgates" },
        "platform": "linkedin",
        "status": "ok",
        "data": {
          "author": { "username": "williamhgates", "followers": 36000000 }
        },
        "cost": 5,
        "fetched_at": "2026-07-04T09:00:03.771Z"
      },
      {
        "index": 2,
        "target": { "platform": "instagram", "handle": "@nosuchaccount" },
        "platform": "instagram",
        "status": "not_found",
        "cost": 0,
        "ext": { "reason": "empty_or_private" },
        "error": {
          "type": "NOT_FOUND",
          "message": "The profile returned no data (private, suspended, or deleted)."
        },
        "fetched_at": "2026-07-04T09:00:02.904Z"
      }
    ],
    "summary": {
      "total": 3,
      "ok": 2,
      "not_found": 1,
      "unsupported": 0,
      "error": 0,
      "deferred": 0,
      "coverage": 0.6666666666666666,
      "credits_charged": 6,
      "credits_refunded": 1
    },
    "legs": []
  },
  "credits_used": 6,
  "credits_remaining": 4521,
  "request_id": "req-abc123",
  "cached": false
}

Rows come back in input order, always all N of them, never paginated. custom_id is echoed verbatim when you supply it, so you can join rows back to your own records without relying on position. legs carries per-upstream call telemetry where the endpoint fans out. comment-lookup returns results and summary only.

Row statuses

Per-item isolation is the point: one dead link never fails the batch. Endpoint-specific detail arrives in ext rather than as new statuses. A caption-less video, for example, is not_found with ext.reason: "no_captions".

The summary

summary is the same nine fields on every fan-out batch, and the same object appears in the SSE done event, so the two can never disagree. coverage is the row ratio ok / total, not a credit ratio. credits_charged always equals the envelope's credits_used.

The YouTube id batches

POST /v1/youtube/videos and POST /v1/youtube/channels are chunk-metered rather than per-row, so they return the ordinary list shape: data.items and data.total, each item identical to the matching single endpoint, with no results or summary block.

They are billed by chunk: 5 credits per 50 ids, charged in full on a 200, because the upstream chunk is consumed whether or not every id in it resolves. A batch that resolves nothing returns 200 with { "items": [], "total": 0 } and is fully refunded, and a total upstream failure is fully refunded too.

Profiles with their latest posts

POST /v1/prism/profiles can return each handle's first page of posts beside its profile. Send "include": "posts" with up to 25 items:

curl -X POST 'https://www.socialcrawl.dev/v1/prism/profiles' \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "include": "posts",
    "since": "2026-09-01",
    "items": [
      { "platform": "tiktok", "handle": "@scout2015" },
      { "platform": "instagram", "handle": "nasa", "stop_at_id": "3990495229280545049" },
      { "platform": "twitch", "handle": "shroud" }
    ]
  }'

Every ok row gains a posts block next to data. The profile in data is unchanged:

{
  "index": 0,
  "target": { "platform": "tiktok", "handle": "@scout2015" },
  "platform": "tiktok",
  "status": "ok",
  "data": { "author": { "username": "scout2015", "followers": 5300 } },
  "posts": {
    "endpoint": "/v1/tiktok/profile/videos",
    "status": "ok",
    "items": [
      /* post rows, exactly as GET /v1/tiktok/profile/videos returns them */
    ],
    "next_cursor": null,
    "has_more": false,
    "stopped_at": "since",
    "credits": 1
  },
  "cost": 2,
  "fetched_at": "2026-09-23T09:00:03.128Z"
}

• endpointstringrequired

  The post list the page came from: /v1/instagram/profile/posts, /v1/tiktok/profile/videos, /v1/youtube/channel/videos, /v1/twitter/user/tweets, /v1/threads/user/posts or /v1/facebook/profile/posts. Continue that handle's walk there, with next_cursor as cursor.

• statusstringrequired

  ok when the page came back with posts, empty when the list had none, failed when the page could not be fetched (the profile row still stands), unsupported when the platform has no post list here, deferred when the 45-second deadline passed first.

• itemsarrayrequired

  The post rows, identical to the single post list endpoint. Empty unless status is ok.

• next_cursorstring | nullrequired

  The cursor for that handle's next page on endpoint.

• has_morebooleanrequired

  true while that handle's list has more pages.

• stopped_atstring | null

  Present only when you sent since or stop_at_id: known_id, since, end or null, as on the single list endpoints.

• creditsintegerrequired

  What the posts page cost: 1 when status is ok, otherwise 0.

since goes on the body and applies to every item. stop_at_id goes on an item and is the id or URL of the newest post you already hold for that handle. Both stop the page exactly as they do on the single list endpoints, so a daily check of 25 creators is one call; see Incremental sync.

A row costs its profile price plus 1 credit for its posts page, and the page is charged only when it came back with posts, including a page that since or stop_at_id then trimmed to nothing. The hold adds one page for every item whose platform has a post list, and every page that was not charged is refunded. A row that is not ok carries no posts block and costs nothing. In the example above the TikTok row costs 2 credits and a Twitch row costs 1, because Twitch answers posts.status: "unsupported".

include accepts only "posts". More than 25 items with it, since or stop_at_id without it, or a since that is not a date is a 400 INVALID_REQUEST and costs nothing. Without include the endpoint behaves exactly as before, with up to 50 items.

Background jobs for thousands of items

When a list is longer than one call takes, POST /v1/prism/jobs runs it in the background. Send up to 5,000 items for prism/profiles or prism/post-stats, in the same item shape that endpoint takes, and read the results when the job finishes:

curl -X POST 'https://www.socialcrawl.dev/v1/prism/jobs' \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: creator-refresh-23-09-2026" \
  -d '{
    "endpoint": "prism/profiles",
    "items": [
      { "platform": "tiktok", "handle": "@scout2015" },
      { "platform": "instagram", "handle": "nasa", "custom_id": "crm-1182" }
    ],
    "webhook": { "url": "https://hooks.yourapp.com/socialcrawl" }
  }'

The response is 202 Accepted with the job:

{
  "success": true,
  "platform": "prism",
  "endpoint": "/v1/prism/jobs",
  "data": {
    "job_id": "bj_7Qk2R9xLpV",
    "endpoint": "prism/profiles",
    "status": "queued",
    "item_count": 2,
    "chunk_count": 1,
    "chunks_done": 0,
    "credits_hold": 2,
    "credits_charged": 0,
    "summary": null,
    "webhook_url": "https://hooks.yourapp.com/socialcrawl",
    "created_at": "2026-09-23T09:00:00.000Z",
    "completed_at": null,
    "expires_at": "2026-09-30T09:00:00.000Z",
    "webhook": {
      "url": "https://hooks.yourapp.com/socialcrawl",
      "secret": "whsec_3f9a…"
    }
  },
  "credits_used": 2,
  "credits_remaining": 4521,
  "request_id": "req-abc123",
  "cached": false
}

Every item is checked before anything is held. A bad item is a 400 INVALID_REQUEST that names its position, and nothing is created or charged.

How a job is billed

The prices are the same as the batch endpoint the job runs. The 202 holds the whole job's worst case, the sum of every item's price, and reports it in credits_used and credits_hold. When the last item finishes, the job settles once. You are charged for the ok rows, and the rest of the hold is refunded. credits_charged on the job then shows the final charge. If your balance cannot cover the hold, the call is a 402 INSUFFICIENT_CREDITS and no job is created. If the job cannot be scheduled, the call is a 502 and the hold is refunded in full. A job whose every row failed upstream ends failed and is charged nothing.

Items run in chunks of 50 (25 with include: "posts"), one chunk after another, and each chunk runs exactly as a call to that endpoint would. A row the chunk's time limit did not reach comes back deferred at 0 credits, and it is not retried: send those items again in a new job.

With an Idempotency-Key, sending the same key with the same body again returns the job it already created, with 200 and 0 credits, and never starts a second job. The same key with a different body is a 422 and nothing is charged.

Reading status and results

GET /v1/prism/jobs/{job_id} is free with the API key that created the job. It returns the job object above without the webhook secret, plus the finished rows:

curl "https://www.socialcrawl.dev/v1/prism/jobs/bj_7Qk2R9xLpV" \
  -H "x-api-key: YOUR_API_KEY"

GET /v1/prism/jobs lists the jobs created with your API key, newest first, 20 per page, and is also free. Send next_cursor back as cursor for the next page.

The completion webhook

With webhook set, the 202 carries webhook.secret (whsec_…). It is shown once, so store it. When the job completes, SocialCrawl sends one POST to your URL with the summary and a link to the results, never the rows:

{
  "event": "batch.completed",
  "job_id": "bj_7Qk2R9xLpV",
  "endpoint": "prism/profiles",
  "status": "completed",
  "summary": {
    "total": 2,
    "ok": 2,
    "not_found": 0,
    "unsupported": 0,
    "error": 0,
    "deferred": 0,
    "coverage": 1,
    "credits_charged": 2,
    "credits_refunded": 0
  },
  "results_url": "https://www.socialcrawl.dev/v1/prism/jobs/bj_7Qk2R9xLpV"
}

The x-socialcrawl-signature header reads t=<unix seconds>,v1=<hex>, where the hex is the HMAC-SHA256 of <t>.<raw body> keyed with your secret. Check it against the raw body, before you parse the JSON:

import { createHmac, timingSafeEqual } from "node:crypto";

function isFromSocialCrawl(rawBody: string, header: string, secret: string) {
  const parts = Object.fromEntries(
    header.split(",").map((part) => part.split("=", 2)),
  );
  const expected = createHmac("sha256", secret)
    .update(`${parts.t}.${rawBody}`)
    .digest("hex");
  const given = Buffer.from(parts.v1 ?? "", "hex");
  const wanted = Buffer.from(expected, "hex");
  return given.length === wanted.length && timingSafeEqual(given, wanted);
}

A delivery can be retried, so it can arrive more than once. Use job_id to handle each job once, then read the rows from results_url with your API key.

Billing, worked

POST /v1/youtube/videos with 1000 ids: ceil(1000 / 50) = 20 chunks, 20 × 5 = 100 credits held upfront. If the upstream returns anything at all, that is the settled charge. If it resolves zero items, all 100 come back.

POST /v1/prism/profiles with 40 TikTok handles and 10 LinkedIn handles: the hold is 40 × 1 + 10 × 5 = 90 credits. If 3 TikTok handles are private and 1 LinkedIn handle is gone, the settled charge is 37 × 1 + 9 × 5 = 82, and 8 credits are refunded, visible as credits_refunded: 8 in the summary.

The pattern is the same everywhere: hold the worst case, settle to the rows that succeeded, refund the difference. You never pay for a not_found, unsupported, error, or deferred row.

HTTP status codes

A 502 still carries the per-row detail, so you can see which items failed and why rather than getting a bare error.

Idempotency and retrying the tail

Every batch endpoint honours an optional Idempotency-Key request header, checked before the credit hold. Keys live for 24 hours.

That makes the natural retry workflow safe:

The 45-second deadline

The fan-out endpoints run under a 45-second budget. Items still queued when it expires come back deferred at 0 credits, and their upstream was never touched. This is a pressure valve, not an error. Re-request the deferred items and they will run.

If you routinely see deferred rows, either lower the item count per call or stream the response so you can start processing rows the moment they land.

Streaming a batch

post-stats, comment-lookup, profiles, and transcripts stream when you send Accept: text/event-stream:

curl -N -X POST 'https://www.socialcrawl.dev/v1/prism/post-stats' \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Accept: text/event-stream" \
  -d '{"urls": ["https://www.youtube.com/watch?v=dQw4w9WgXcQ"]}'

Each row is emitted the moment it settles, as a result event keyed by row type (stat, item, profile, or transcript), followed by a terminal done event carrying the same nine-field summary as the sync response. Billing is identical either way. An SSE batch is stored for idempotent replay as its sync-JSON equivalent, so a retry with the same key replays whichever wire format it asks for. See Streaming (SSE) for the full frame contract.

The upfront hold is settled through one serialized lifecycle shared by normal completion, timeout, error, and disconnect recovery. If an immediate refund fails, the terminal state keeps the outstanding committed charge and queues that exact remainder for automatic compensation; a late executor cannot race the recovery path into a duplicate refund.

POST /v1/youtube/videos and POST /v1/youtube/channels do not stream.

When to use batch instead

Next steps