# Batch endpoints (/docs/batch) Six POST endpoints that take an array and return one row per item, with per-item isolation, per-row billing, and automatic refunds for the rows that did not resolve. Six endpoints take a whole array in one `POST` and return one row per input item. Use them when you need the same lookup for a list: a morning refresh of 400 post URLs, 60 handles to vet, 100 transcripts for a corpus. Batch is not a different product. Each row is byte-identical to what the matching single endpoint returns. You trade N requests for one, and you pay for the rows that actually resolved. ## The six endpoints | Endpoint | Cap per call | Body array | Billing | | ------------------------------- | ------------ | ---------- | ---------------------------------------------------------------------------------------------------------- | | `POST /v1/prism/post-stats` | 100 URLs | `urls` | Per successful URL at that platform's rate (1 credit on most, 2 for Instagram, 5 for LinkedIn post stats) | | `POST /v1/prism/comment-lookup` | 25 items | `items` | Per `found` item (TikTok 2, Instagram 5; `deep_scan` raises it). Whole batch holds at most 100 credits | | `POST /v1/prism/profiles` | 50 items, 25 with posts | `items` | Per successful row at each platform's tier (most 1, LinkedIn 5), plus 1 credit per page of posts with `include: "posts"` | | `POST /v1/youtube/transcripts` | 100 ids | `ids` | 3 credits per successful transcript | | `POST /v1/youtube/videos` | 1000 ids | `ids` | 5 credits per 50-id chunk, `ceil(ids / 50) × 5` | | `POST /v1/youtube/channels` | 1000 ids | `ids` | 5 credits per 50-id chunk, `ceil(ids / 50) × 5` | Authentication is the same `x-api-key` header as every other endpoint. These responses are never cached, so `cached` is always `false`. A batch refresh exists to read the live number. However many items it carries, a batch counts as **one** request against the 600 per minute and 50 concurrent budgets. See [Rate limits](/docs/rate-limits.md). ## Making a request Send a JSON body. The request body ceiling is **1 MB**. A larger body is rejected with `413 PAYLOAD_TOO_LARGE` before anything is parsed or charged. ```bash title="cURL" curl -X POST 'https://www.socialcrawl.dev/v1/prism/profiles' \ -H "x-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -H "Idempotency-Key: vetting-batch-1" \ -d '{ "items": [ { "platform": "tiktok", "handle": "@scout2015" }, { "platform": "linkedin", "handle": "williamhgates", "custom_id": "vet-1" }, { "platform": "instagram", "handle": "@nosuchaccount" } ] }' ``` ```ts title="TypeScript" const BASE = "https://www.socialcrawl.dev"; const res = await fetch(`${BASE}/v1/prism/profiles`, { method: "POST", headers: { "x-api-key": process.env.SOCIALCRAWL_API_KEY!, "Content-Type": "application/json", "Idempotency-Key": "vetting-batch-1", }, body: JSON.stringify({ items: [ { platform: "tiktok", handle: "@scout2015" }, { platform: "linkedin", handle: "williamhgates", custom_id: "vet-1" }, { platform: "instagram", handle: "@nosuchaccount" }, ], }), }); const { data } = await res.json(); console.log(data.summary.ok, data.summary.credits_charged); ``` ```python title="Python" import os import requests res = requests.post( "https://www.socialcrawl.dev/v1/prism/profiles", headers={ "x-api-key": os.environ["SOCIALCRAWL_API_KEY"], "Idempotency-Key": "vetting-batch-1", }, json={ "items": [ {"platform": "tiktok", "handle": "@scout2015"}, {"platform": "linkedin", "handle": "williamhgates", "custom_id": "vet-1"}, {"platform": "instagram", "handle": "@nosuchaccount"}, ] }, ) summary = res.json()["data"]["summary"] print(summary["ok"], summary["credits_charged"]) ``` Items are **not** de-duplicated. Two identical URLs are two rows and two charges. De-duplicate before you send, and use `Idempotency-Key` to make retries safe. ## The response The four fan-out endpoints (`post-stats`, `comment-lookup`, `profiles`, `transcripts`) return per-row results plus a summary: ```json title="Response" { "success": true, "platform": "prism", "endpoint": "/v1/prism/profiles", "data": { "results": [ { "index": 0, "target": { "platform": "tiktok", "handle": "@scout2015" }, "platform": "tiktok", "status": "ok", "data": { "author": { "username": "scout2015", "followers": 5300 } }, "cost": 1, "fetched_at": "2026-07-04T09:00:03.128Z" }, { "index": 1, "custom_id": "vet-1", "target": { "platform": "linkedin", "handle": "williamhgates" }, "platform": "linkedin", "status": "ok", "data": { "author": { "username": "williamhgates", "followers": 36000000 } }, "cost": 5, "fetched_at": "2026-07-04T09:00:03.771Z" }, { "index": 2, "target": { "platform": "instagram", "handle": "@nosuchaccount" }, "platform": "instagram", "status": "not_found", "cost": 0, "ext": { "reason": "empty_or_private" }, "error": { "type": "NOT_FOUND", "message": "The profile returned no data (private, suspended, or deleted)." }, "fetched_at": "2026-07-04T09:00:02.904Z" } ], "summary": { "total": 3, "ok": 2, "not_found": 1, "unsupported": 0, "error": 0, "deferred": 0, "coverage": 0.6666666666666666, "credits_charged": 6, "credits_refunded": 1 }, "legs": [] }, "credits_used": 6, "credits_remaining": 4521, "request_id": "req-abc123", "cached": false } ``` Rows come back **in input order**, always all N of them, never paginated. `custom_id` is echoed verbatim when you supply it, so you can join rows back to your own records without relying on position. `legs` carries per-upstream call telemetry where the endpoint fans out. `comment-lookup` returns `results` and `summary` only. ### Row statuses | `status` | Meaning | Charged | | ------------- | --------------------------------------------------------------------------- | ------- | | `ok` | Item fetched, `data` present. `comment-lookup` calls this `found`. | Yes | | `not_found` | Upstream authoritatively 404'd, deleted, private, or never existed. | No | | `unsupported` | The item could not be routed to any endpoint (unknown host or path shape). | No | | `error` | Upstream or internal failure after retries; `error.type` explains it. | No | | `deferred` | The 45-second batch deadline hit before this item started. Never attempted. | No | Per-item isolation is the point: one dead link never fails the batch. Endpoint-specific detail arrives in `ext` rather than as new statuses. A caption-less video, for example, is `not_found` with `ext.reason: "no_captions"`. ### The summary `summary` is the same nine fields on every fan-out batch, and the same object appears in the SSE `done` event, so the two can never disagree. `coverage` is the **row** ratio `ok / total`, not a credit ratio. `credits_charged` always equals the envelope's `credits_used`. ### The YouTube id batches `POST /v1/youtube/videos` and `POST /v1/youtube/channels` are chunk-metered rather than per-row, so they return the ordinary list shape: `data.items` and `data.total`, each item identical to the matching single endpoint, with no `results` or `summary` block. They are billed by chunk: 5 credits per 50 ids, charged in full on a 200, because the upstream chunk is consumed whether or not every id in it resolves. A batch that resolves **nothing** returns `200` with `{ "items": [], "total": 0 }` and is fully refunded, and a total upstream failure is fully refunded too. ## Profiles with their latest posts `POST /v1/prism/profiles` can return each handle's first page of posts beside its profile. Send `"include": "posts"` with up to 25 items: ```bash title="cURL" curl -X POST 'https://www.socialcrawl.dev/v1/prism/profiles' \ -H "x-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "include": "posts", "since": "2026-09-01", "items": [ { "platform": "tiktok", "handle": "@scout2015" }, { "platform": "instagram", "handle": "nasa", "stop_at_id": "3990495229280545049" }, { "platform": "twitch", "handle": "shroud" } ] }' ``` Every `ok` row gains a `posts` block next to `data`. The profile in `data` is unchanged: ```json title="Response" { "index": 0, "target": { "platform": "tiktok", "handle": "@scout2015" }, "platform": "tiktok", "status": "ok", "data": { "author": { "username": "scout2015", "followers": 5300 } }, "posts": { "endpoint": "/v1/tiktok/profile/videos", "status": "ok", "items": [ /* post rows, exactly as GET /v1/tiktok/profile/videos returns them */ ], "next_cursor": null, "has_more": false, "stopped_at": "since", "credits": 1 }, "cost": 2, "fetched_at": "2026-09-23T09:00:03.128Z" } ``` | Field | Type | Required | Description | | --- | --- | --- | --- | | `endpoint` | `string` | yes | The post list the page came from: /v1/instagram/profile/posts, /v1/tiktok/profile/videos, /v1/youtube/channel/videos, /v1/twitter/user/tweets, /v1/threads/user/posts or /v1/facebook/profile/posts. Continue that handle's walk there, with next_cursor as cursor. | | `status` | `string` | yes | ok when the page came back with posts, empty when the list had none, failed when the page could not be fetched (the profile row still stands), unsupported when the platform has no post list here, deferred when the 45-second deadline passed first. | | `items` | `array` | yes | The post rows, identical to the single post list endpoint. Empty unless status is ok. | | `next_cursor` | `string \| null` | yes | The cursor for that handle's next page on endpoint. | | `has_more` | `boolean` | yes | true while that handle's list has more pages. | | `stopped_at` | `string \| null` | no | Present only when you sent since or stop_at_id: known_id, since, end or null, as on the single list endpoints. | | `credits` | `integer` | yes | What the posts page cost: 1 when status is ok, otherwise 0. | `since` goes on the body and applies to every item. `stop_at_id` goes on an item and is the id or URL of the newest post you already hold for that handle. Both stop the page exactly as they do on the single list endpoints, so a daily check of 25 creators is one call; see [Incremental sync](/docs/pagination.md#incremental-sync). A row costs its profile price plus 1 credit for its posts page, and the page is charged only when it came back with posts, including a page that `since` or `stop_at_id` then trimmed to nothing. The hold adds one page for every item whose platform has a post list, and every page that was not charged is refunded. A row that is not `ok` carries no `posts` block and costs nothing. In the example above the TikTok row costs 2 credits and a Twitch row costs 1, because Twitch answers `posts.status: "unsupported"`. `include` accepts only `"posts"`. More than 25 items with it, `since` or `stop_at_id` without it, or a `since` that is not a date is a `400 INVALID_REQUEST` and costs nothing. Without `include` the endpoint behaves exactly as before, with up to 50 items. ## Background jobs for thousands of items When a list is longer than one call takes, `POST /v1/prism/jobs` runs it in the background. Send up to 5,000 items for `prism/profiles` or `prism/post-stats`, in the same item shape that endpoint takes, and read the results when the job finishes: ```bash title="cURL" curl -X POST 'https://www.socialcrawl.dev/v1/prism/jobs' \ -H "x-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -H "Idempotency-Key: creator-refresh-23-09-2026" \ -d '{ "endpoint": "prism/profiles", "items": [ { "platform": "tiktok", "handle": "@scout2015" }, { "platform": "instagram", "handle": "nasa", "custom_id": "crm-1182" } ], "webhook": { "url": "https://hooks.yourapp.com/socialcrawl" } }' ``` The response is `202 Accepted` with the job: ```json title="Response" { "success": true, "platform": "prism", "endpoint": "/v1/prism/jobs", "data": { "job_id": "bj_7Qk2R9xLpV", "endpoint": "prism/profiles", "status": "queued", "item_count": 2, "chunk_count": 1, "chunks_done": 0, "credits_hold": 2, "credits_charged": 0, "summary": null, "webhook_url": "https://hooks.yourapp.com/socialcrawl", "created_at": "2026-09-23T09:00:00.000Z", "completed_at": null, "expires_at": "2026-09-30T09:00:00.000Z", "webhook": { "url": "https://hooks.yourapp.com/socialcrawl", "secret": "whsec_3f9a…" } }, "credits_used": 2, "credits_remaining": 4521, "request_id": "req-abc123", "cached": false } ``` | Body field | Required | Meaning | | ---------- | -------- | ------- | | `endpoint` | yes | `prism/profiles` or `prism/post-stats`. | | `items` | yes | 1 to 5,000 items. For `prism/profiles`, `{ platform, handle, custom_id?, stop_at_id? }`. For `prism/post-stats`, `{ url, custom_id? }`. | | `include` | no | `"posts"`, `prism/profiles` only. Adds each handle's first page of posts, as described above. | | `since` | no | A date or ISO 8601 timestamp, `prism/profiles` with `include: "posts"` only. | | `webhook` | no | `{ "url": "https://…" }`, a public https URL to call when the job completes. | Every item is checked before anything is held. A bad item is a `400 INVALID_REQUEST` that names its position, and nothing is created or charged. ### How a job is billed The prices are the same as the batch endpoint the job runs. The `202` holds the whole job's worst case, the sum of every item's price, and reports it in `credits_used` and `credits_hold`. When the last item finishes, the job settles once. You are charged for the `ok` rows, and the rest of the hold is refunded. `credits_charged` on the job then shows the final charge. If your balance cannot cover the hold, the call is a `402 INSUFFICIENT_CREDITS` and no job is created. If the job cannot be scheduled, the call is a `502` and the hold is refunded in full. A job whose every row failed upstream ends `failed` and is charged nothing. Items run in chunks of 50 (25 with `include: "posts"`), one chunk after another, and each chunk runs exactly as a call to that endpoint would. A row the chunk's time limit did not reach comes back `deferred` at 0 credits, and it is not retried: send those items again in a new job. With an `Idempotency-Key`, sending the same key with the same body again returns the job it already created, with `200` and 0 credits, and never starts a second job. The same key with a different body is a `422` and nothing is charged. ### Reading status and results `GET /v1/prism/jobs/{job_id}` is free with the API key that created the job. It returns the job object above without the webhook secret, plus the finished rows: ```bash title="cURL" curl "https://www.socialcrawl.dev/v1/prism/jobs/bj_7Qk2R9xLpV" \ -H "x-api-key: YOUR_API_KEY" ``` | Field | Meaning | | ----- | ------- | | `status` | `queued`, `running`, `completed` or `failed`. | | `chunks_done`, `chunk_count` | Progress. | | `summary` | The same nine-field summary as a batch call, or `null` until the job completes. | | `results` | The rows of up to five finished chunks, each exactly as the batch endpoint returns it, with `index` set to its position in the items you submitted. | | `next_cursor` | Send it back as `cursor` for the next rows. `null` at the end, and also while the next chunk is still running. | | `results_complete` | `true` once every row has been read. | | `expires_at` | The job and its results are kept for 7 days. | `GET /v1/prism/jobs` lists the jobs created with your API key, newest first, 20 per page, and is also free. Send `next_cursor` back as `cursor` for the next page. ### The completion webhook With `webhook` set, the `202` carries `webhook.secret` (`whsec_…`). It is shown once, so store it. When the job completes, SocialCrawl sends one `POST` to your URL with the summary and a link to the results, never the rows: ```json title="JSON" { "event": "batch.completed", "job_id": "bj_7Qk2R9xLpV", "endpoint": "prism/profiles", "status": "completed", "summary": { "total": 2, "ok": 2, "not_found": 0, "unsupported": 0, "error": 0, "deferred": 0, "coverage": 1, "credits_charged": 2, "credits_refunded": 0 }, "results_url": "https://www.socialcrawl.dev/v1/prism/jobs/bj_7Qk2R9xLpV" } ``` The `x-socialcrawl-signature` header reads `t=,v1=`, where the hex is the HMAC-SHA256 of `.` keyed with your secret. Check it against the raw body, before you parse the JSON: ```ts title="TypeScript" import { createHmac, timingSafeEqual } from "node:crypto"; function isFromSocialCrawl(rawBody: string, header: string, secret: string) { const parts = Object.fromEntries( header.split(",").map((part) => part.split("=", 2)), ); const expected = createHmac("sha256", secret) .update(`${parts.t}.${rawBody}`) .digest("hex"); const given = Buffer.from(parts.v1 ?? "", "hex"); const wanted = Buffer.from(expected, "hex"); return given.length === wanted.length && timingSafeEqual(given, wanted); } ``` A delivery can be retried, so it can arrive more than once. Use `job_id` to handle each job once, then read the rows from `results_url` with your API key. ## Billing, worked `POST /v1/youtube/videos` with 1000 ids: `ceil(1000 / 50) = 20` chunks, 20 × 5 = **100 credits held upfront**. If the upstream returns anything at all, that is the settled charge. If it resolves zero items, all 100 come back. `POST /v1/prism/profiles` with 40 TikTok handles and 10 LinkedIn handles: the hold is `40 × 1 + 10 × 5 = 90` credits. If 3 TikTok handles are private and 1 LinkedIn handle is gone, the settled charge is `37 × 1 + 9 × 5 = 82`, and 8 credits are refunded, visible as `credits_refunded: 8` in the summary. The pattern is the same everywhere: hold the worst case, settle to the rows that succeeded, refund the difference. You never pay for a `not_found`, `unsupported`, `error`, or `deferred` row. ## HTTP status codes | Condition | Status | | ---------------------------------------------------- | ------------------------------------------------------------- | | At least one row succeeded | `200` | | Zero rows succeeded but at least one `error` | `502` with `data.results` still present, full refund | | Zero rows succeeded, all `not_found` / `unsupported` | `200` with `coverage: 0`, charged per model (usually nothing) | | Body invalid, array empty, or over the cap | `400 INVALID_REQUEST`, nothing charged | | Balance cannot cover the hold | `402 INSUFFICIENT_CREDITS`, nothing charged | | Body over 1 MB | `413 PAYLOAD_TOO_LARGE`, nothing parsed or charged | A `502` still carries the per-row detail, so you can see which items failed and why rather than getting a bare error. ## Idempotency and retrying the tail Every batch endpoint honours an optional `Idempotency-Key` request header, checked **before** the credit hold. Keys live for 24 hours. | Reuse | Outcome | | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | Same key, same body | The stored response replays with `X-Idempotent-Replay: true`, `Idempotent-Replayed: true`, and `X-Credits-Used: 0`. Nothing runs again, nothing is charged again | | Same key, different body | `422`. The key is bound to the exact request it first saw | | Same key on another account | Independent claim; keys are account-scoped | That makes the natural retry workflow safe: Read the summary. `deferred` and `error` counts tell you how much of the batch is outstanding. Build a **new** request containing only those rows, and give it a **new** key. Send it. Retrying the whole batch under the original key replays, so retrying only the tail costs only the tail. ## The 45-second deadline The fan-out endpoints run under a 45-second budget. Items still queued when it expires come back `deferred` at 0 credits, and their upstream was never touched. This is a pressure valve, not an error. Re-request the deferred items and they will run. If you routinely see `deferred` rows, either lower the item count per call or [stream the response](/docs/streaming.md) so you can start processing rows the moment they land. ## Streaming a batch `post-stats`, `comment-lookup`, `profiles`, and `transcripts` stream when you send `Accept: text/event-stream`: ```bash title="cURL" curl -N -X POST 'https://www.socialcrawl.dev/v1/prism/post-stats' \ -H "x-api-key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -H "Accept: text/event-stream" \ -d '{"urls": ["https://www.youtube.com/watch?v=dQw4w9WgXcQ"]}' ``` Each row is emitted the moment it settles, as a `result` event keyed by row type (`stat`, `item`, `profile`, or `transcript`), followed by a terminal `done` event carrying the same nine-field summary as the sync response. Billing is identical either way. An SSE batch is stored for idempotent replay as its sync-JSON equivalent, so a retry with the same key replays whichever wire format it asks for. See [Streaming (SSE)](/docs/streaming.md) for the full frame contract. The upfront hold is settled through one serialized lifecycle shared by normal completion, timeout, error, and disconnect recovery. If an immediate refund fails, the terminal state keeps the outstanding committed charge and queues that exact remainder for automatic compensation; a late executor cannot race the recovery path into a duplicate refund. `POST /v1/youtube/videos` and `POST /v1/youtube/channels` do not stream. ## When to use batch instead | You have | Use | | ------------------------------------------ | ---------------------------------------------------------------------- | | One item | The single GET endpoint. Batch has no advantage and a heavier response | | A list of the same lookup | Batch. One request slot, one round trip, per-row billing | | Different lookups combined into one answer | A [Prism composite](/docs/prism.md), which fans out across endpoints | | Thousands of the same lookup | A [background job](#background-jobs-for-thousands-of-items), up to 5,000 items | | The same lookup on a schedule | A [Monitor](/docs/webhooks.md), delivered to you by webhook | ## Next steps - [Streaming (SSE)](/docs/streaming.md): Per-row frames and the terminal done event. - [Endpoint pricing](/docs/endpoint-pricing.md): The exact per-endpoint figures behind per-row costs. - [Errors](/docs/errors.md): Every error.type value a row can carry. - [Choosing endpoints](/docs/choosing-endpoints.md): Picking between the single, batch, and composite forms.