Error Handling

Every failure returns the same JSON envelope: a stable error.type to branch on, an HTTP status, and a doc_url pointing at that code's entry on this page. Six failures are worth retrying. credits_used always reports the charge committed at response time; automatic compensation covers the rare case where an immediate refund cannot commit.

{
  "success": false,
  "error": {
    "type": "INSUFFICIENT_CREDITS",
    "message": "Your account has 0 credits remaining. This endpoint requires 1 credits.",
    "status": 402,
    "doc_url": "https://www.socialcrawl.dev/docs/errors#insufficient-credits"
  },
  "credits_used": 0,
  "credits_remaining": 0,
  "request_id": "req-abc123"
}

Three fields carry the debugging weight:

• error.type is what you branch on. It is stable for a given failure and never changes meaning under /v1.
• credits_used is the net committed charge for the failed request. It is 0 when no deduction occurred or the refund committed. A non-zero value on a retryable failure means the immediate refund did not commit and automatic compensation is pending; retain request_id.
• request_id matches the X-Request-Id header, which is present on every response, success or error. Include it when you contact support so we can trace the exact request in our logs.

The doc_url is an anchor on this page, so you can jump straight to a code with #insufficient-credits or #upstream-error.

Error codes

Which errors are safe to retry

The retryable failures are IDEMPOTENCY_IN_PROGRESS, RATE_LIMITED, CONCURRENCY_LIMIT, UPSTREAM_ERROR, SERVICE_UNAVAILABLE, and INTERNAL_ERROR.

Every other client error is deterministic. IDEMPOTENCY_IN_PROGRESS is the one retryable 409: wait for Retry-After, then send the same key and payload again. See Handling retries for copy-paste backoff loops.

Error reference

Each code below carries a stable anchor, the exact target of the doc_url in the error envelope.

MISSING_API_KEY

401. No x-api-key header was present on the request. Add the header with one of your keys from Dashboard → API Keys. No credits are deducted.

INVALID_API_KEY

401. The key is malformed, not found, revoked, or expired. Check the key in Dashboard → API Keys. No credits are deducted.

INSUFFICIENT_CREDITS

402. Your balance is lower than the endpoint's cost. Top up in Dashboard → Billing. No credits are deducted. credits_remaining reports your current balance.

KEY_BUDGET_EXCEEDED

402. The API key you used has its own credit limit, and this request would take it past that limit. Your account balance is untouched and nothing was deducted.

This is not the same problem as INSUFFICIENT_CREDITS, and topping up will not fix it. A per-key limit is a cap you set yourself so that one key (typically a test, CI, or staging key) can never spend more than a fixed amount of the shared account balance. Hitting it means that key did its job.

To resolve it, open Dashboard → API Keys and either raise the key's limit, reset its usage counter back to zero, or remove the limit entirely. You can also switch to a key with no limit set, which is how production keys are usually configured.

credits_remaining still reports your account balance, so a large number there alongside this error is expected. It is the clearest signal that the cap, not the balance, is what stopped the request.

{
  "success": false,
  "error": {
    "type": "KEY_BUDGET_EXCEEDED",
    "message": "This API key has a limit of 500 credits and has used 498. This endpoint requires 5 credits. Your account balance (12480 credits) was not charged. Raise or reset this key's limit in Dashboard → API Keys, or use a key with no limit.",
    "status": 402,
    "doc_url": "https://www.socialcrawl.dev/docs/errors#key-budget-exceeded"
  },
  "credits_used": 0,
  "credits_remaining": 12480,
  "request_id": "req-abc123"
}

Do not retry this one on a timer. The counter is cumulative rather than windowed, so it never clears on its own. Only a limit change or a usage reset will let the key spend again. See Authentication.

INVALID_REQUEST

400. A required parameter is missing, a value failed format validation, or no oneOf member was satisfied. Check the endpoint's required parameters in the API Reference. No credits are deducted.

Some rejections carry error.details you can branch on. It has two shapes and they never appear together, so branch on which key is present. All of these rejections run before billing, so a rejected request costs 0 credits.

METHOD_NOT_ALLOWED

405. The method is not accepted on this path. Registry read paths advertise Allow: GET, HEAD; POST-only routes advertise Allow: POST. The JSON body and X-Request-Id are still canonical. No credits are deducted.

The exceptions are the six batch endpoints listed under Idempotency, which take POST, and the /v1/web/* job surface, which takes POST on job, monitor, and session creation and PATCH or DELETE on monitors, sessions, and jobs.

ENDPOINT_NOT_FOUND

404. The platform or resource is not supported. Check the API Reference for the correct path. No credits are deducted.

RESOURCE_NOT_FOUND

404. The upstream returned 404, or the resource exists but contains no usable data, for example a nonexistent handle. Verify the handle or URL. Credits are refunded when this is triggered by an empty upstream body.

Two endpoint families narrow the cause with error.details.reason, so you can tell a permanent gap from a fixable mistake.

PAYLOAD_TOO_LARGE

413. The request's JSON body exceeded the 1 MB size limit and was rejected before parsing, so no credits were deducted. Reduce the batch size, for example by splitting a large ids or urls array across several requests.

IDEMPOTENCY_KEY_CONFLICT

409. The Idempotency-Key conflicts with an incompatible active reservation. Generate a fresh key, ideally a UUIDv4. Keys are scoped per account; this error does not reveal or depend on another account's keys. No credits are deducted.

IDEMPOTENCY_IN_PROGRESS

409. Another request with this key and the same payload is still running. This retry was not dispatched or charged. Wait for Retry-After: 1, then retry with the same key. Once the original finishes, the retry replays its stored result at 0 credits.

IDEMPOTENCY_REPLAY_UNAVAILABLE

409. The original request completed and its billing outcome is known, but its response exceeded the 64 KB idempotent replay storage limit. This retry was not dispatched or charged. The same key cannot reconstruct the original body. Use a new key only if you intend to make and pay for a new request.

IDEMPOTENCY_KEY_PAYLOAD_MISMATCH

422. You reused an Idempotency-Key with a different payload, meaning different query parameters on a GET or a different JSON body on a POST. Use a new key for the new payload. No credits are deducted.

COHORT_LIMIT_EXCEEDED

400. The account already holds the maximum of 100 cohorts under this API key. Delete an unused cohort (DELETE /v1/cohorts/{cohortId}) before creating another.

COHORT_MEMBER_LIMIT_EXCEEDED

400. The member upload would exceed the cohort limit. Reduce the upload or remove members before retrying.

COHORT_IDENTITY_PLATFORM_UNSUPPORTED

400. The supplied identity platform is not supported for cohort queries. Use a supported platform.

COHORT_IDENTITY_CONFLICT

409. The normalized platform identity is already assigned to another external ID in this cohort. Use the existing external ID or a different identity.

COHORT_QUERY_NOT_CANCELLABLE

409. The query is already terminal and cannot be cancelled. Read its status or results instead.

COHORT_QUERY_NOT_READY

409. The query has not completed, so results are not available yet. Wait for a terminal status before reading results.

COHORT_RESULT_TOO_LARGE

413. One decrypted result exceeds the 1,000,000-byte response-page ceiling. No partial result is returned.

RATE_LIMITED

429. More than 600 requests in a 1-minute sliding window on the same API key. Slow down and retry after Retry-After. The response carries X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset. No credits are deducted. See Rate Limits.

CONCURRENCY_LIMIT

429. More than 50 concurrent requests were in flight on the same API key. Reduce concurrency and retry. No credits are deducted. The response carries X-Concurrency-Limit and X-Concurrency-Remaining, not X-RateLimit-*. See Rate Limits.

UPSTREAM_ERROR

502. The upstream platform returned an error, after our own automatic retries. Retry after a short backoff. Credits are refunded.

To tell an upstream outage from a transient blip, check the live status page or GET /v1/status for the platform's current circuit state.

SERVICE_UNAVAILABLE

503. Several causes. An open circuit breaker or an upstream provider throttle carries Retry-After: 30. If the idempotency claim store is temporarily unavailable, a keyed request fails closed before billing with Retry-After: 1; retry with the same key. No balance lookup is attempted on that path, so credits_remaining is null and X-Credits-Remaining is omitted.

A 503 without a Retry-After header is not transient. It means the endpoint has been withdrawn because its upstream no longer serves it, and the message names the reason. Retrying will never succeed, so branch on the presence of Retry-After rather than on the status alone. Nothing is deducted in the first place on a withdrawn endpoint, since the router rejects it before billing, so there is no refund to wait for. Check the changelog for the replacement.

INTERNAL_ERROR

500. An unexpected error occurred on our side. Retry, and if it persists, contact support with the request_id. The server attempts the refund before responding. If it cannot commit, credits_used remains non-zero and the exact outstanding remainder is queued for automatic compensation rather than falsely reported as refunded.

Idempotency

Send an Idempotency-Key header to make a charging request safe to retry. The key is honored on every charging endpoint: the GET /v1/* catch-all and all six POST endpoints (/v1/prism/post-stats, /v1/prism/comment-lookup, /v1/prism/profiles, /v1/youtube/transcripts, /v1/youtube/videos, /v1/youtube/channels).

• A retry with the same key and the same payload replays the stored terminal status and payload, charges nothing (X-Credits-Used: 0, X-Idempotent-Replay: true), and does not re-run the upstream call. Replay billing metadata is refreshed: the current balance is returned when known.
• The key's identity covers the request payload: query parameters for GET, and the canonical JSON body for POST. Reusing a key with a different payload returns IDEMPOTENCY_KEY_PAYLOAD_MISMATCH (422) rather than a false replay.
• Keys are scoped per account. Two accounts can use the same opaque key independently.
• A same-key request that is still running returns IDEMPOTENCY_IN_PROGRESS (409) with Retry-After: 1. It is not dispatched or charged; retry with the same key.
• An unfinished claim expires after 5 minutes so a crashed request cannot lock the key indefinitely.
• A completed outcome whose body exceeded 64 KB returns IDEMPOTENCY_REPLAY_UNAVAILABLE (409). Use a new key only to make a new billable request.
• If the claim store is unavailable, the keyed request fails closed at 503 with Retry-After: 1 before billing.
• Completed outcomes are retained for 24 hours. Use a fresh UUIDv4 per logical operation.
• GET SSE cannot be combined with Idempotency-Key; the unsupported combination returns an unbilled 400 INVALID_REQUEST before dispatch. POST batch streams keep sync-equivalent replay.

Refund rules

Credits are refunded automatically when:

• Upstream returns a 5xx error (UPSTREAM_ERROR, 502).
• The circuit breaker rejects the request, or the upstream provider rate-limits us (SERVICE_UNAVAILABLE, 503).
• An unexpected server error occurs (INTERNAL_ERROR, 500).
• Upstream returns 200 with an empty body, for example a nonexistent handle, triggering RESOURCE_NOT_FOUND (404).

Credits are never deducted for these outcomes, so no refund is needed:

• Cache hits (X-Cache: HIT, X-Credits-Used: 0).
• METHOD_NOT_ALLOWED (405), IDEMPOTENCY_KEY_CONFLICT (409), IDEMPOTENCY_IN_PROGRESS (409), IDEMPOTENCY_REPLAY_UNAVAILABLE (409), and IDEMPOTENCY_KEY_PAYLOAD_MISMATCH (422).
• KEY_BUDGET_EXCEEDED (402). The request is refused before the charge lands, and the key's own usage counter is left where it was.
• Idempotent replays. The original charge already appears on your earlier request.

Client errors (MISSING_API_KEY, INVALID_API_KEY, INVALID_REQUEST, ENDPOINT_NOT_FOUND) are rejected before any deduction. See Credits for the full refund matrix.

Handling retries

Retry only the errors marked retryable above: IDEMPOTENCY_IN_PROGRESS (409), RATE_LIMITED or CONCURRENCY_LIMIT (429), and transient 500, 502, or 503 responses. Honor Retry-After: idempotency contention and claim-store outages send 1, circuit/provider 503s send 30, rate-limit 429s send the seconds until reset, and concurrency 429s send a short hint.

Otherwise back off exponentially with full jitter, so a pool of your own workers does not resynchronize and stampede the cap together. Cap the attempts and surface the error rather than retrying forever. Never retry a 4xx client error other than 429: the request is deterministic and will fail the same way.

Retryable failures attempt an immediate refund; if that database mutation fails, exact compensation is queued and the response keeps a non-zero committed credits_used. Add an Idempotency-Key header (see Idempotency) so a network-level retry of a successful sync call replays rather than re-runs.

Each loop below retries only the transient statuses (429, 500, 502, 503, 504), honors Retry-After when present, falls back to exponential backoff with jitter, and gives up after five attempts.

#!/usr/bin/env bash
url="https://www.socialcrawl.dev/v1/tiktok/profile?handle=charlidamelio"
max_retries=5
attempt=0

while :; do
  body=$(mktemp); headers=$(mktemp)
  status=$(curl -s -o "$body" -D "$headers" -w '%{http_code}' \
    -H "x-api-key: $SOCIALCRAWL_API_KEY" "$url")

  if [ "$status" -lt 400 ]; then
    cat "$body"; rm -f "$body" "$headers"; break
  fi

  case "$status" in
    429|500|502|503|504) ;;  # retryable, fall through
    *) echo "Non-retryable $status"; cat "$body"; rm -f "$body" "$headers"; exit 1 ;;
  esac

  attempt=$((attempt + 1))
  if [ "$attempt" -gt "$max_retries" ]; then
    echo "Gave up after $max_retries retries (last status $status)"
    rm -f "$body" "$headers"; exit 1
  fi

  # Retry-After (seconds) wins; else exponential backoff 2^attempt with jitter.
  retry_after=$(grep -i '^retry-after:' "$headers" | tr -d '\r' | awk '{print $2}')
  if [ -n "$retry_after" ]; then
    delay="$retry_after"
  else
    delay=$(awk "BEGIN{srand(); print (2 ^ $attempt) * (0.5 + rand() / 2)}")
  fi
  rm -f "$body" "$headers"
  sleep "$delay"
done

import os
import random
import time

import requests  # pip install requests

BASE_URL = "https://www.socialcrawl.dev/v1"
RETRYABLE = {429, 500, 502, 503, 504}


def get_with_retry(path, params=None, max_retries=5):
    headers = {"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]}
    for attempt in range(max_retries + 1):
        resp = requests.get(f"{BASE_URL}{path}", params=params, headers=headers)
        # Success, or a client error we must not retry: hand it back / raise.
        if resp.ok or resp.status_code not in RETRYABLE or attempt == max_retries:
            resp.raise_for_status()
            return resp.json()
        # Retry-After (seconds) wins; else exponential backoff with full jitter.
        retry_after = resp.headers.get("Retry-After")
        delay = float(retry_after) if retry_after else (2 ** attempt) * (0.5 + random.random() / 2)
        time.sleep(delay)


data = get_with_retry("/tiktok/profile", {"handle": "charlidamelio"})
print(data)

// Node 18+ (built-in fetch). No dependencies.
const BASE_URL = "https://www.socialcrawl.dev/v1";
const RETRYABLE = new Set([429, 500, 502, 503, 504]);
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

async function getWithRetry(path, { params = {}, maxRetries = 5 } = {}) {
  const url = new URL(BASE_URL + path);
  for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
  const headers = { "x-api-key": process.env.SOCIALCRAWL_API_KEY };

  for (let attempt = 0; ; attempt++) {
    const res = await fetch(url, { headers });
    if (res.ok || !RETRYABLE.has(res.status)) {
      if (!res.ok)
        throw new Error(`SocialCrawl ${res.status}: ${await res.text()}`);
      return res.json();
    }
    if (attempt >= maxRetries) {
      throw new Error(
        `Gave up after ${maxRetries} retries (last status ${res.status})`,
      );
    }
    // Retry-After (seconds) wins; else exponential backoff with full jitter.
    const retryAfter = Number(res.headers.get("Retry-After"));
    const delayMs =
      Number.isFinite(retryAfter) && retryAfter > 0
        ? retryAfter * 1000
        : 2 ** attempt * 1000 * (0.5 + Math.random() / 2);
    await sleep(delayMs);
  }
}

const data = await getWithRetry("/tiktok/profile", {
  params: { handle: "charlidamelio" },
});
console.log(data);

Debugging tips

• Every error envelope carries a doc_url pointing at the matching section on this page, for example https://www.socialcrawl.dev/docs/errors#insufficient-credits.
• request_id matches the X-Request-Id header and the request_id column on Dashboard → Activity Logs.
• Persistent 502s on a known-good input usually indicate an upstream outage. Check the status page, or GET /v1/status, for the platform's circuit state.
• A high retry count on a slow-but-successful call is surfaced in the X-Upstream-Retries response header.

Next steps