Async requests

Some calls take longer than your client waits. youtube/channel/about, the transcript endpoints and twitter/ai-search can each run for 30 seconds or more, while the MCP server and many HTTP clients stop waiting at 30 seconds. Any GET endpoint can run in the background instead: you get a job straight away, and you read the finished response when it is ready.

The call is the same call. It is validated before the job is created, then served, cached, billed and refunded exactly as it would be if you had waited for it.

Run one GET in the background

Add the header Prefer: respond-async (RFC 7240) to the call you already make:

curl "https://www.socialcrawl.dev/v1/youtube/channel/about?handle=@mkbhd" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Prefer: respond-async"

If your client cannot set a header (an agent's fetch tool, a no-code HTTP step), add async=true to the query instead:

curl "https://www.socialcrawl.dev/v1/youtube/channel/about?handle=@mkbhd&async=true" \
  -H "x-api-key: YOUR_API_KEY"

The answer is 202 Accepted with the job. The Location header is the URL to poll, Retry-After says how many seconds to wait first, and Preference-Applied: respond-async confirms the header was honoured:

{
  "success": true,
  "platform": "prism",
  "endpoint": "/v1/prism/jobs",
  "data": {
    "job_id": "bj_Vn3kQ8sLx2",
    "endpoint": "youtube/channel/about",
    "billing": "per_call",
    "status": "queued",
    "item_count": 1,
    "chunk_count": 1,
    "chunks_done": 0,
    "credits_hold": 0,
    "credits_charged": 0,
    "credits_max": 25,
    "summary": null,
    "webhook_url": null,
    "results_url": "https://www.socialcrawl.dev/v1/prism/jobs/bj_Vn3kQ8sLx2",
    "created_at": "2026-10-24T09:00:00.000Z",
    "completed_at": null,
    "expires_at": "2026-10-31T09:00:00.000Z"
  },
  "credits_used": 0,
  "credits_remaining": 4521,
  "request_id": "req-abc123",
  "cached": false
}

async=false or async=0 is an ordinary call. Any other value is a free 400, so a typo never runs a different call. dry_run=1 still wins: the call is validated and quoted for 0 credits and no job is created.

Read the result

Poll GET /v1/prism/jobs/{job_id} (the Location URL). Reading a job is free:

curl "https://www.socialcrawl.dev/v1/prism/jobs/bj_Vn3kQ8sLx2" \
  -H "x-api-key: YOUR_API_KEY"

While the job runs, status is queued or running and the answer carries Retry-After. Once it is completed, results[0].response is exactly the body the GET would have returned, envelope and all:

{
  "success": true,
  "data": {
    "job_id": "bj_Vn3kQ8sLx2",
    "endpoint": "youtube/channel/about",
    "billing": "per_call",
    "status": "completed",
    "credits_charged": 25,
    "summary": { "total": 1, "ok": 1, "not_found": 0, "unsupported": 0, "error": 0, "deferred": 0, "coverage": 1, "credits_charged": 25, "credits_refunded": 0 },
    "results": [
      {
        "index": 0,
        "status": "ok",
        "http_status": 200,
        "params": { "handle": "@mkbhd" },
        "credits_used": 25,
        "response": { "success": true, "data": { "...": "the channel, as the GET returns it" }, "credits_used": 25 }
      }
    ],
    "next_cursor": null,
    "results_complete": true
  },
  "credits_used": 0
}

A minimal poll loop:

const headers = { "x-api-key": process.env.SOCIALCRAWL_API_KEY! };

const submit = await fetch(
  "https://www.socialcrawl.dev/v1/youtube/channel/about?handle=@mkbhd",
  { headers: { ...headers, Prefer: "respond-async" } },
);
const jobUrl = submit.headers.get("location")!;

for (;;) {
  const res = await fetch(jobUrl, { headers });
  const { data } = await res.json();
  if (data.status === "completed" || data.status === "failed") {
    console.log(data.results[0].response);
    break;
  }
  const wait = Number(res.headers.get("retry-after") ?? "5");
  await new Promise((r) => setTimeout(r, wait * 1000));
}

Run many calls to one GET as a job

To run the same endpoint for a list, send the queries to POST /v1/prism/jobs with the endpoint's path. Each item is one call's query parameters:

curl -X POST "https://www.socialcrawl.dev/v1/prism/jobs" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "endpoint": "youtube/channel/about",
    "items": [
      { "handle": "@mkbhd" },
      { "handle": "@veritasium" },
      { "channelId": "UCX6OQ3DkcsbYNE6H8uQQuVA" }
    ],
    "webhook": { "url": "https://hooks.yourapp.com/socialcrawl" }
  }'

• endpoint is any GET endpoint's path, with or without the leading /v1/.
• items holds 1 to 1,000 objects. A number or boolean value is sent as text, and a list is sent comma-separated.
• Every item is validated as that GET would be before the job is stored. A bad item is a free 400 that names its position, for example Item 2: Missing required parameter: handle, and nothing is created.
• Items run 3 at a time, each under the same time limit as a direct call, and results page five chunks at a time through next_cursor.
• webhook works as it does for batch jobs: one signed batch.completed call with the summary and the results URL.

How an async call is billed

An async GET holds nothing when you submit it, and credits_used on the 202 is 0. Each item is billed as that GET when it runs, at the same price, with the same cache and the same refunds, and credits_charged on the job adds them up. credits_max on the 202 is the most the job can cost.

If your balance cannot cover one call, the submit is the same 402 INSUFFICIENT_CREDITS the direct call would give, and no job is created. A job whose every call failed ends failed.

Each item carries a fixed idempotency key, so if a worker restarts mid-job, an item that already ran is replayed, not billed twice. Send your own Idempotency-Key on the submit and a retry returns the same job instead of creating a second one.

When to use it

Two endpoints work differently. prism/discover has its own background runs: send its own async=true, and the router leaves both the parameter and Prefer to it. prism/answers only answers as a stream, so it cannot run as a job: call it with Accept: text/event-stream.

Jobs and their results are kept for 7 days. GET /v1/prism/jobs lists your key's jobs, newest first.