Error Handling

All errors follow the same envelope format:

{
  "success": false,
  "error": {
    "type": "INSUFFICIENT_CREDITS",
    "message": "Your account has 0 credits remaining. This endpoint requires 1 credits.",
    "status": 402,
    "doc_url": "https://www.socialcrawl.dev/docs/errors#insufficient-credits"
  },
  "credits_used": 0,
  "credits_remaining": 0,
  "request_id": "req-abc123"
}

Every error envelope carries credits_used — the net credits charged for the failed request. Error paths deduct then refund, so this is 0 in effectively every case. Read it to confirm a failure cost you nothing rather than inferring it from the refund rules below.

Every response — success or error — includes an X-Request-Id header matching request_id. Include it when contacting support so we can trace the exact request in our logs.

The doc_url on every error points at the matching section on this page. Jump straight to a code with its anchor, e.g. #insufficient-credits or #upstream-error.

Error Codes

Only the five transient failures (RATE_LIMITED, CONCURRENCY_LIMIT, UPSTREAM_ERROR, SERVICE_UNAVAILABLE, INTERNAL_ERROR) are safe to retry. Every client error (4xx other than 429) is deterministic: the same request will fail the same way, so retrying wastes time and credits are never at stake because none were deducted. See Handling retries for copy-paste backoff loops.

Error Code Reference

Each code below carries a stable anchor — the exact target of the doc_url in the error envelope.

MISSING_API_KEY

401 — No x-api-key header was present on the request. Add the header with one of your keys from Dashboard → API Keys. No credits are deducted.

INVALID_API_KEY

401 — The key is malformed, not found, revoked, or expired. Check the key in Dashboard → API Keys. No credits are deducted.

INSUFFICIENT_CREDITS

402 — Your balance is lower than the endpoint's cost. Top up in Dashboard → Billing. No credits are deducted. credits_remaining reports your current balance.

KEY_BUDGET_EXCEEDED

402 — The API key you used has its own credit limit, and this request would take it past that limit. Your account balance is untouched and nothing was deducted.

This is not the same problem as INSUFFICIENT_CREDITS, and topping up will not fix it. A per-key limit is a cap you set yourself so that one key — typically a test, CI, or staging key — can never spend more than a fixed amount of the shared account balance. Hitting it means that key did its job.

To resolve it, open Dashboard → API Keys and either raise the key's limit, reset its usage counter back to zero, or remove the limit entirely. You can also switch to a key with no limit set (production keys usually have none).

credits_remaining still reports your account balance, so a large number there alongside this error is expected and is the clearest signal that the cap — not the balance — is what stopped the request:

{
  "success": false,
  "error": {
    "type": "KEY_BUDGET_EXCEEDED",
    "message": "This API key has a limit of 500 credits and has used 498. This endpoint requires 5 credits. Your account balance (12480 credits) was not charged — raise or reset this key's limit in Dashboard → API Keys, or use a key with no limit.",
    "status": 402,
    "doc_url": "https://www.socialcrawl.dev/docs/errors#key-budget-exceeded"
  },
  "credits_used": 0,
  "credits_remaining": 12480,
  "request_id": "req-abc123"
}

Do not retry this one on a timer. The counter is cumulative rather than windowed, so it never clears on its own — only a limit change or a usage reset will let the key spend again. See Authentication.

INVALID_REQUEST

400 — A required parameter is missing, a value failed format validation, or no oneOf member was satisfied. Check the endpoint's required parameters in the API Reference. No credits are deducted.

Some rejections carry an error.details.reason you can branch on:

Both checks run before billing, so a mangled query costs you nothing.

This most often bites on non-Latin search queries. A query that is silently corrupted in transit cannot match anything, and it used to come back as an empty 200, which looked like "this platform has no content in my language". It is now a 400 that names the fault:

{
  "success": false,
  "error": {
    "type": "INVALID_REQUEST",
    "message": "Parameter 'query' is not valid UTF-8. It contains Unicode replacement characters (U+FFFD), which almost always means the client sent the value in a non-UTF-8 charset (e.g. windows-1251 or latin-1). Send the value as UTF-8.",
    "details": { "reason": "invalid_utf8" }
  }
}

A correctly-encoded non-Latin query needs no special handling. Cyrillic, Greek, Arabic, Korean, and Japanese all work as-is:

curl -G https://www.socialcrawl.dev/v1/threads/search \
  -H "x-api-key: YOUR_KEY" \
  --data-urlencode "query=работа"

error.details.did_you_mean — the self-correcting rejections

A second family of INVALID_REQUEST carries error.details.did_you_mean instead of a reason: the exact parameter, or param=value pair, to send instead. You get it when you send a foreign pagination param (did_you_mean: "cursor" — see Pagination), when you combine two options the upstream cannot honor together, or when an option you sent needs a companion parameter you left out. A few of these also carry remove_params, a list of parameters to drop; apply both fields and you land on a valid request.

So error.details has two shapes and they never appear together — branch on which key is present. These rejections are all pre-billing, so they cost 0 credits, and the hint is stable enough for an SDK to auto-correct against.

METHOD_NOT_ALLOWED

405 — The method is not accepted on this path. Most of the /v1/* surface is GET-only, and those paths answer any other method with this error plus an Allow: GET header. The exceptions are the six batch endpoints listed under Idempotency, which take POST, and the /v1/web/* job surface, which takes POST on job, monitor, and session creation and PATCH/DELETE on monitors, sessions, and jobs. No credits are deducted.

ENDPOINT_NOT_FOUND

404 — The platform or resource is not supported. Check the API Reference for the correct path. No credits are deducted.

RESOURCE_NOT_FOUND

404 — The upstream returned 404, or the resource exists but contains no usable data (e.g. a nonexistent handle). Verify the handle or URL. Credits are refunded when this is triggered by an empty upstream body.

Transcript endpoints (e.g. /v1/youtube/video/transcript) additionally set error.details.reason so you can tell exactly why no transcript came back:

All four are deterministic for a given video — retrying the same URL returns the same result and never costs credits. Around 5–10% of public YouTube videos have no retrievable captions, so batch pipelines should expect and skip these rather than retry them.

The batch POST /v1/youtube/transcripts endpoint reports these per row as status: "not_found" with ext.reason, and refunds each affected row. Its vocabulary is coarser: only no_captions (which also covers captions-disabled and login-gated videos) and video_gone. If you need to tell a captions-disabled video from a genuinely caption-less one, re-fetch that id through the singular endpoint.

COHORT_LIMIT_EXCEEDED

400 — The account already holds the maximum of 100 cohorts under this API key. Delete an unused cohort (DELETE /v1/cohorts/{cohortId}) before creating another.

COHORT_MEMBER_LIMIT_EXCEEDED

400 — The member upload would exceed the cohort limit. Reduce the upload or remove members before retrying.

COHORT_IDENTITY_PLATFORM_UNSUPPORTED

400 — The supplied identity platform is not supported for cohort queries. Use a supported platform.

COHORT_IDENTITY_CONFLICT

409 — The normalized platform identity is already assigned to another external ID in this cohort. Use the existing external ID or a different identity.

COHORT_QUERY_NOT_CANCELLABLE

409 — The query is already terminal and cannot be cancelled. Read its status or results instead.

COHORT_QUERY_NOT_READY

409 — The query has not completed, so results are not available yet. Wait for a terminal status before reading results.

COHORT_RESULT_TOO_LARGE

413 — One decrypted result exceeds the 1,000,000-byte response-page ceiling. No partial result is returned.

IDEMPOTENCY_KEY_CONFLICT

409 — The Idempotency-Key you sent is already in use by another account. Generate a fresh key (a UUIDv4 is recommended). No credits are deducted.

IDEMPOTENCY_KEY_PAYLOAD_MISMATCH

422 — You reused an Idempotency-Key with a different payload — different query parameters on a GET, or a different JSON body on a POST. Use a new key for the new payload. No credits are deducted.

PAYLOAD_TOO_LARGE

413: The request's JSON body exceeded the 1 MB size limit and was rejected before parsing, so no credits were deducted. Reduce the batch size, for example by splitting a large ids or urls array across several requests.

RATE_LIMITED

429 — More than 600 requests in a 1-minute sliding window on the same API key. Slow down and retry after Retry-After. Response carries X-RateLimit-Limit/Remaining/Reset. No credits are deducted. See Rate Limits.

CONCURRENCY_LIMIT

429 — More than 50 concurrent requests were in flight on the same API key. Reduce concurrency and retry. No credits are deducted. Response carries X-Concurrency-Limit/Remaining (not X-RateLimit-*). See Rate Limits.

UPSTREAM_ERROR

502 — The upstream platform returned an error (after our automatic retries). Retry after a short backoff. Credits are refunded. To tell an upstream outage from a transient blip, check the live status page or GET /v1/status for the platform's current circuit state.

SERVICE_UNAVAILABLE

503 — Several causes. The two common ones are transient and both carry Retry-After: 30: the circuit breaker is open after repeated upstream failures, or the upstream data provider is momentarily rate-limiting us. Wait 30 seconds and retry. Credits are refunded.

A 503 without a Retry-After header is not transient. It means the endpoint has been withdrawn because its upstream no longer serves it, and the message names the reason. Retrying will never succeed, so branch on the presence of Retry-After rather than on the status alone. Nothing is deducted in the first place on a withdrawn endpoint — the router rejects it before billing — so there is no refund to wait for. Check the changelog for the replacement.

INTERNAL_ERROR

500 — An unexpected error occurred on our side. Retry; if it persists, contact support with the request_id. Credits are refunded.

Idempotency

Send an Idempotency-Key header to make a charging request safe to retry. The key is honored on every charging endpoint — the GET /v1/* catch-all and all six POST endpoints (/v1/prism/post-stats, /v1/prism/comment-lookup, /v1/prism/profiles, /v1/youtube/transcripts, /v1/youtube/videos, /v1/youtube/channels).

• A retry with the same key and the same payload replays the stored response verbatim, charges nothing (X-Credits-Used: 0, X-Idempotent-Replay: true), and does not re-run the upstream call — so a network retry or a cron double-fire never double-charges or double-creates.
• The key's identity covers the request payload: query parameters for GET, and the canonical JSON body for POST. Reusing a key with a different payload returns IDEMPOTENCY_KEY_PAYLOAD_MISMATCH (422) rather than a false replay.
• A key already claimed by another account returns IDEMPOTENCY_KEY_CONFLICT (409).
• Keys are retained for 24 hours. Use a fresh UUIDv4 per logical operation.

Refund Rules

Credits are automatically refunded when:

• Upstream returns a 5xx error (UPSTREAM_ERROR, 502)
• The circuit breaker rejects the request, or the upstream provider rate-limits us (SERVICE_UNAVAILABLE, 503)
• An unexpected server error occurs (INTERNAL_ERROR, 500)
• Upstream returns 200 with an empty body — e.g. a nonexistent handle — triggering RESOURCE_NOT_FOUND (404)

Credits are never deducted for these outcomes, so no refund is needed:

• Cache hits (X-Cache: HIT, X-Credits-Used: 0)
• METHOD_NOT_ALLOWED (405), IDEMPOTENCY_KEY_CONFLICT (409), and IDEMPOTENCY_KEY_PAYLOAD_MISMATCH (422)
• KEY_BUDGET_EXCEEDED (402) — the request is refused before the charge lands, and the key's own usage counter is left where it was
• Idempotent replays — the original charge already appears on your earlier request

Client errors (MISSING_API_KEY, INVALID_API_KEY, INVALID_REQUEST, ENDPOINT_NOT_FOUND) are rejected before any deduction.

Handling retries

Retry only the statuses marked retryable above: 429 (RATE_LIMITED or CONCURRENCY_LIMIT), 500, 502, and 503. When a response carries a Retry-After header (a 503 always sends Retry-After: 30; a rate-limit 429 sends seconds until the window resets; a concurrency 429 sends a short hint), wait that long before the first retry. Otherwise back off exponentially with full jitter so a pool of your own workers does not resynchronize and stampede the cap together. Cap the attempts and surface the error rather than retrying forever. Never retry a 4xx client error other than 429: the request is deterministic and will fail the same way.

Because credits are refunded on every retryable failure, a retry loop never double-charges. Add an Idempotency-Key header (see Idempotency) if you also want a network-level retry of a successful call to replay rather than re-run.

#!/usr/bin/env bash
# Retry only transient failures (429, 500, 502, 503, 504). Honor Retry-After
# when present, else exponential backoff with jitter. Give up after 5 tries.
url="https://www.socialcrawl.dev/v1/tiktok/profile?handle=charlidamelio"
max_retries=5
attempt=0

while :; do
  body=$(mktemp); headers=$(mktemp)
  status=$(curl -s -o "$body" -D "$headers" -w '%{http_code}' \
    -H "x-api-key: $SOCIALCRAWL_API_KEY" "$url")

  if [ "$status" -lt 400 ]; then
    cat "$body"; rm -f "$body" "$headers"; break
  fi

  case "$status" in
    429|500|502|503|504) ;;  # retryable, fall through
    *) echo "Non-retryable $status"; cat "$body"; rm -f "$body" "$headers"; exit 1 ;;
  esac

  attempt=$((attempt + 1))
  if [ "$attempt" -gt "$max_retries" ]; then
    echo "Gave up after $max_retries retries (last status $status)"
    rm -f "$body" "$headers"; exit 1
  fi

  # Retry-After (seconds) wins; else exponential backoff 2^attempt with jitter.
  retry_after=$(grep -i '^retry-after:' "$headers" | tr -d '\r' | awk '{print $2}')
  if [ -n "$retry_after" ]; then
    delay="$retry_after"
  else
    delay=$(awk "BEGIN{srand(); print (2 ^ $attempt) * (0.5 + rand() / 2)}")
  fi
  rm -f "$body" "$headers"
  sleep "$delay"
done

import os
import random
import time

import requests  # pip install requests

BASE_URL = "https://www.socialcrawl.dev/v1"
RETRYABLE = {429, 500, 502, 503, 504}


def get_with_retry(path, params=None, max_retries=5):
    headers = {"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]}
    for attempt in range(max_retries + 1):
        resp = requests.get(f"{BASE_URL}{path}", params=params, headers=headers)
        # Success, or a client error we must not retry: hand it back / raise.
        if resp.ok or resp.status_code not in RETRYABLE or attempt == max_retries:
            resp.raise_for_status()
            return resp.json()
        # Retry-After (seconds) wins; else exponential backoff with full jitter.
        retry_after = resp.headers.get("Retry-After")
        delay = float(retry_after) if retry_after else (2 ** attempt) * (0.5 + random.random() / 2)
        time.sleep(delay)


data = get_with_retry("/tiktok/profile", {"handle": "charlidamelio"})
print(data)

// Node 18+ (built-in fetch). No dependencies.
const BASE_URL = "https://www.socialcrawl.dev/v1";
const RETRYABLE = new Set([429, 500, 502, 503, 504]);
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

async function getWithRetry(path, { params = {}, maxRetries = 5 } = {}) {
  const url = new URL(BASE_URL + path);
  for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
  const headers = { "x-api-key": process.env.SOCIALCRAWL_API_KEY };

  for (let attempt = 0; ; attempt++) {
    const res = await fetch(url, { headers });
    if (res.ok || !RETRYABLE.has(res.status)) {
      if (!res.ok)
        throw new Error(`SocialCrawl ${res.status}: ${await res.text()}`);
      return res.json();
    }
    if (attempt >= maxRetries) {
      throw new Error(
        `Gave up after ${maxRetries} retries (last status ${res.status})`,
      );
    }
    // Retry-After (seconds) wins; else exponential backoff with full jitter.
    const retryAfter = Number(res.headers.get("Retry-After"));
    const delayMs =
      Number.isFinite(retryAfter) && retryAfter > 0
        ? retryAfter * 1000
        : 2 ** attempt * 1000 * (0.5 + Math.random() / 2);
    await sleep(delayMs);
  }
}

const data = await getWithRetry("/tiktok/profile", {
  params: { handle: "charlidamelio" },
});
console.log(data);

Debugging Tips

• Every error envelope carries a doc_url pointing at the matching section on this page, e.g. https://www.socialcrawl.dev/docs/errors#insufficient-credits
• request_id matches the X-Request-Id header and the request_id column on Dashboard → Activity Logs
• Persistent 502s on a known-good input usually indicate an upstream outage — check the status page (or GET /v1/status) for the platform's circuit state
• A high retry count on a slow-but-successful call is surfaced in the X-Upstream-Retries response header