SocialCrawl

Error Handling

Every SocialCrawl error code with its status, retry verdict, and refund behavior, plus copy-paste backoff loops

Every failure returns the same JSON envelope: a stable error.type to branch on, an HTTP status, an error.retryable flag, and a doc_url pointing at that code's entry on this page. Six failures are worth retrying. credits_used always reports the charge committed at response time; automatic compensation covers the rare case where an immediate refund cannot commit.

Response
{
  "success": false,
  "error": {
    "type": "INSUFFICIENT_CREDITS",
    "message": "Your account has 0 credits remaining. This endpoint requires 1 credits. Retrying this request will not succeed until your account balance changes.",
    "status": 402,
    "doc_url": "https://www.socialcrawl.dev/docs/errors#insufficient-credits",
    "retryable": false,
    "details": {
      "reason": "balance_too_low",
      "retry_will_succeed": false,
      "next_step": "Top up your credits on the billing page (or turn on auto-recharge there), then send the request again.",
      "top_up_url": "https://www.socialcrawl.dev/dashboard/billing/payments",
      "credits_required": 1
    }
  },
  "credits_used": 0,
  "credits_remaining": 0,
  "request_id": "req-abc123"
}

Four fields carry the debugging weight:

  • error.type is what you branch on. It is stable for a given failure and never changes meaning under /v1.
  • error.retryable is true when sending the identical request again can succeed, and false when it will fail the same way until you change something. It matches the Retryable column in the table below, so a client can decide without keeping its own list of codes.
  • credits_used is the net committed charge for the failed request. It is 0 when no deduction occurred or the refund committed. A non-zero value on a retryable failure means the immediate refund did not commit and automatic compensation is pending; retain request_id.
  • request_id matches the X-Request-Id header, which is present on every response, success or error. Include it when you contact support so we can trace the exact request in our logs.

The doc_url is an anchor on this page, so you can jump straight to a code with #insufficient-credits or #upstream-error.

Error codes

CodeStatusRetryableRefundWhat it means
MISSING_API_KEY401NoNot chargedNo x-api-key header on the request
INVALID_API_KEY401NoNot chargedKey is malformed, not found, revoked, or expired
INSUFFICIENT_CREDITS402NoNot chargedBalance is lower than the endpoint cost
KEY_BUDGET_EXCEEDED402NoNot chargedThis key has spent its own per-key credit limit. The account balance is untouched
INVALID_REQUEST400NoNot chargedMissing, malformed, conflicting, or non-empty undeclared parameters
METHOD_NOT_ALLOWED405NoNot chargedThe method is not accepted on this path
ENDPOINT_NOT_FOUND404NoNot chargedPlatform or resource is not supported
RESOURCE_NOT_FOUND404NoRefundedUpstream returned 404, or the resource exists but contains no usable data
PAYLOAD_TOO_LARGE413NoNot chargedThe JSON request body exceeds the 1 MB size limit, rejected before parsing
IDEMPOTENCY_KEY_CONFLICT409NoNot chargedThe key conflicts with an incompatible active reservation
IDEMPOTENCY_IN_PROGRESS409Yes, Retry-AfterNot chargedAnother request with the same key and payload is still running
IDEMPOTENCY_REPLAY_UNAVAILABLE409NoNot chargedThe completed response exceeded the replay limit (160 KB compressed)
IDEMPOTENCY_KEY_PAYLOAD_MISMATCH422NoNot chargedYou reused an Idempotency-Key with different query parameters or request body
COHORT_LIMIT_EXCEEDED400NoNot chargedThe account already holds the maximum of 100 cohorts
COHORT_MEMBER_LIMIT_EXCEEDED400NoNot chargedA cohort member upload would exceed the cohort limit
COHORT_IDENTITY_PLATFORM_UNSUPPORTED400NoNot chargedThe identity platform is not supported for cohort queries
COHORT_IDENTITY_CONFLICT409NoNot chargedA normalized identity is already assigned to another external ID
COHORT_QUERY_NOT_CANCELLABLE409NoNot chargedThe cohort query is terminal and cannot be cancelled
COHORT_QUERY_NOT_READY409NoNot chargedThe cohort query has not completed
COHORT_RESULT_TOO_LARGE413NoNot chargedOne cohort result exceeds the response-page byte ceiling
RATE_LIMITED429Yes, after backoffNot chargedMore than 600 requests in a 1-minute window on the same API key
CONCURRENCY_LIMIT429Yes, after backoffNot chargedMore than 50 concurrent requests on the same API key
UPSTREAM_ERROR502, 504Yes, with backoffRefundedUpstream platform returned an error, or the call reached the platform time limit
SERVICE_UNAVAILABLE503Yes, with backoffRefundedCircuit breaker open, upstream provider throttling us, or a withdrawn endpoint
INTERNAL_ERROR500Yes, with backoffRefundedUnexpected error on our side
RUN_NOT_FOUND404NoNot chargedNo discover run with this id belongs to your account
RUN_STATE_CONFLICT409NoNot chargedThe discover run's state does not allow this call (for example resuming a completed run)
RUN_EXPIRED410NoNot chargedThe discover run's retention passed and its rows were deleted
NEEDS_ASYNC422NoNot chargedA discover request with async=false is over what one call can answer
CURSOR_EXPIRED410NoNot chargedAn events feed cursor is older than the 30-day retention
SCHEDULE_NOT_FOUND404NoNot chargedNo discover schedule with this id belongs to your account

Which errors are safe to retry

The retryable failures are IDEMPOTENCY_IN_PROGRESS, RATE_LIMITED, CONCURRENCY_LIMIT, UPSTREAM_ERROR, SERVICE_UNAVAILABLE, and INTERNAL_ERROR. Every error body says the same thing in error.retryable, so you can branch on that field instead of copying this list. The flag follows the code, so a withdrawn endpoint's SERVICE_UNAVAILABLE still reads true; tell that one apart by its missing Retry-After header, as described under SERVICE_UNAVAILABLE. The one response that overrides the code's flag is described under UPSTREAM_ERROR.

Every other client error is deterministic. IDEMPOTENCY_IN_PROGRESS is the one retryable 409: wait for Retry-After, then send the same key and payload again. See Handling retries for copy-paste backoff loops.

Error reference

Each code below carries a stable anchor, the exact target of the doc_url in the error envelope.

MISSING_API_KEY

401. No x-api-key header was present on the request. Add the header with one of your keys from Dashboard → API Keys. No credits are deducted.

INVALID_API_KEY

401. The key is malformed, not found, revoked, or expired. Check the key in Dashboard → API Keys. No credits are deducted.

INSUFFICIENT_CREDITS

402. Your balance is lower than the endpoint's cost. Top up in Dashboard → Billing. No credits are deducted. credits_remaining reports your current balance.

Sending the same request again will not succeed until your balance changes, so keep this code out of your retry loop. The message ends with a sentence saying so, and error.details gives the same answer as fields:

error.details keyWhat it holds
reasonbalance_too_low
retry_will_succeedAlways false. The identical request fails the same way until credits are added, by a top-up or by auto-recharge.
next_stepOne sentence saying what to do next.
top_up_urlThe billing page in the dashboard, where you buy credits or turn on auto-recharge.
credits_requiredThe credits this request needed. Present on GET endpoints.
pricing"ceiling" when the endpoint holds its worst-case cost up front and refunds whatever the call does not use. Absent on fixed-price endpoints.
lower_cost_withSent with pricing: "ceiling": the parameters that shrink the hold.
key_prefixThe prefix of the API key that sent the request, as Dashboard → API Keys shows it, for example sc_...hYFw. Present on GET endpoints.
key_nameThe name you gave that key. Present on GET endpoints.
account_email_maskedThe account's login email with all but its first characters and domain hidden, for example ni***@example.com. Present on GET endpoints.

If your team has more than one account, key_prefix and account_email_masked tell you which one ran out. The message names them as well, in a sentence before the last one.

A 402 carries no Retry-After header. Waiting does not help: only a top-up or auto-recharge changes the answer, and either one takes effect on your next request.

When the number is a ceiling. /v1/threads/search holds 1 credit for every search window its limit allows, plus the query relaxation searches a multi-word query can fan out to, and refunds whatever the call does not use. A balance below that hold gets a 402 even though the finished call might have cost less. When a smaller request would hold less, the 402 says so: details.pricing is "ceiling", details.lower_cost_with is ["limit", "expand"], and the message explains that a smaller limit, or expand=false, holds fewer credits.

Response
{
  "success": false,
  "error": {
    "type": "INSUFFICIENT_CREDITS",
    "message": "Your account has 3 credits remaining. This endpoint requires 11 credits. The 11 credits required are the most this call can cost: the credits it holds for search windows and query relaxation that it does not use are refunded afterwards. A smaller `limit`, or `expand=false`, holds fewer credits. Retrying this request will not succeed until your account balance changes.",
    "status": 402,
    "doc_url": "https://www.socialcrawl.dev/docs/errors#insufficient-credits",
    "retryable": false,
    "details": {
      "reason": "balance_too_low",
      "retry_will_succeed": false,
      "next_step": "Top up your credits on the billing page (or turn on auto-recharge there), then send the request again.",
      "top_up_url": "https://www.socialcrawl.dev/dashboard/billing/payments",
      "credits_required": 11,
      "pricing": "ceiling",
      "lower_cost_with": ["limit", "expand"]
    }
  },
  "credits_used": 0,
  "credits_remaining": 3,
  "request_id": "req-abc123"
}

Stopping a runaway client. If one script or agent spends the balance that your other keys depend on, give its key a per-key credit limit. That key then stops at its own cap with KEY_BUDGET_EXCEEDED, and the rest of the account keeps working.

KEY_BUDGET_EXCEEDED

402. The API key you used has its own credit limit, and this request would take it past that limit. Your account balance is untouched and nothing was deducted.

This is not the same problem as INSUFFICIENT_CREDITS, and topping up will not fix it. A per-key limit is a cap you set yourself so that one key (typically a test, CI, or staging key) can never spend more than a fixed amount of the shared account balance. Hitting it means that key did its job.

To resolve it, open Dashboard → API Keys and either raise the key's limit, reset its usage counter back to zero, or remove the limit entirely. You can also switch to a key with no limit set, which is how production keys are usually configured.

credits_remaining still reports your account balance, so a large number there alongside this error is expected. It is the clearest signal that the cap, not the balance, is what stopped the request.

Response
{
  "success": false,
  "error": {
    "type": "KEY_BUDGET_EXCEEDED",
    "message": "This API key has a limit of 500 credits and has used 498. This endpoint requires 5 credits. Your account balance (12480 credits) was not charged. Raise or reset this key's limit in Dashboard → API Keys, or use a key with no limit. Retrying this request will not succeed until this key's credit limit is raised, removed or reset.",
    "status": 402,
    "doc_url": "https://www.socialcrawl.dev/docs/errors#key-budget-exceeded",
    "retryable": false,
    "details": {
      "reason": "key_budget_reached",
      "retry_will_succeed": false,
      "next_step": "Raise or remove this key's credit limit, or reset its usage, on the API keys page, or send the request with a key that has no limit.",
      "key_settings_url": "https://www.socialcrawl.dev/dashboard/api/keys",
      "credits_required": 5
    }
  },
  "credits_used": 0,
  "credits_remaining": 12480,
  "request_id": "req-abc123"
}

error.details has the same keys as INSUFFICIENT_CREDITS, with two differences: reason is key_budget_reached, and key_settings_url (the API keys page) replaces top_up_url, because buying credits does not lift a key's own limit.

Do not retry this one on a timer. The counter is cumulative rather than windowed, so it never clears on its own. Only a limit change or a usage reset will let the key spend again. See Authentication.

INVALID_REQUEST

400. A required parameter is missing, a value failed format validation, or no oneOf member was satisfied. Check the endpoint's required parameters in the API Reference. A rejected request costs 0 credits.

Most rejections run before billing, so nothing is deducted. A few can only be detected by the data source after the call has been charged, such as asking for a page past the end of a search. Those are refunded in full before the response is sent, so credits_used is still 0.

Some rejections carry error.details you can branch on. It has two shapes and they never appear together, so branch on which key is present.

Getting 200 with an empty items array on a correctly-encoded query is a different thing entirely, and is not an error. Most platform search endpoints are keyword indexes rather than semantic ones, so a full sentence tends to return nothing in any language (find a job returns results; i cannot find a job returns none). Search with keywords, not sentences.

METHOD_NOT_ALLOWED

405. The method is not accepted on this path. Registry read paths advertise Allow: GET, HEAD; POST-only routes advertise Allow: POST. The JSON body and X-Request-Id are still canonical. No credits are deducted.

The exceptions are the six batch endpoints listed under Idempotency, which take POST, and the /v1/web/* job surface, which takes POST on job, monitor, and session creation and PATCH or DELETE on monitors, sessions, and jobs.

ENDPOINT_NOT_FOUND

404. The platform or resource is not supported. Check the API Reference for the correct path. No credits are deducted.

RESOURCE_NOT_FOUND

404. The upstream returned 404, or the resource exists but contains no usable data, for example a nonexistent handle. Verify the handle or URL. The deduction is refunded in full, so credits_used is 0.

Three endpoint families narrow the cause with error.details.reason, so you can tell a permanent gap from a fixable mistake.

PAYLOAD_TOO_LARGE

413. The request's JSON body exceeded the 1 MB size limit and was rejected before parsing, so no credits were deducted. Reduce the batch size, for example by splitting a large ids or urls array across several requests.

IDEMPOTENCY_KEY_CONFLICT

409. The Idempotency-Key conflicts with an incompatible active reservation. Generate a fresh key, ideally a UUIDv4. Keys are scoped per account; this error does not reveal or depend on another account's keys. No credits are deducted.

IDEMPOTENCY_IN_PROGRESS

409. Another request with this key and the same payload is still running. This retry was not dispatched or charged. Wait for the Retry-After seconds, then retry with the same key. The wait is 1 second while the original is young and grows to a quarter of the time it has been running, up to 30, so a long call is not polled every second. Once the original finishes, the retry replays its stored result at 0 credits.

IDEMPOTENCY_REPLAY_UNAVAILABLE

409. The original request completed and its billing outcome is known, but its response exceeded the idempotent replay storage limit: a response over 64 KB is stored compressed and replays normally, unless it is still over 160 KB compressed. This retry was not dispatched or charged. The same key cannot reconstruct the original body. Send the request with a new key to get the data again: inside the endpoint's cache window (freshness.cache_ttl_s on /v1/utility/endpoint) it is served from the cache at 0 credits, and after the window it is a new billable request.

IDEMPOTENCY_KEY_PAYLOAD_MISMATCH

422. You reused an Idempotency-Key with a different payload, meaning different query parameters on a GET or a different JSON body on a POST. Use a new key for the new payload. No credits are deducted.

COHORT_LIMIT_EXCEEDED

400. The account already holds the maximum of 100 cohorts under this API key. Delete an unused cohort (DELETE /v1/cohorts/{cohortId}) before creating another.

COHORT_MEMBER_LIMIT_EXCEEDED

400. The member upload would exceed the cohort limit. Reduce the upload or remove members before retrying.

COHORT_IDENTITY_PLATFORM_UNSUPPORTED

400. The supplied identity platform is not supported for cohort queries. Use a supported platform.

COHORT_IDENTITY_CONFLICT

409. The normalized platform identity is already assigned to another external ID in this cohort. Use the existing external ID or a different identity.

COHORT_QUERY_NOT_CANCELLABLE

409. The query is already terminal and cannot be cancelled. Read its status or results instead.

COHORT_QUERY_NOT_READY

409. The query has not completed, so results are not available yet. Wait for a terminal status before reading results.

COHORT_RESULT_TOO_LARGE

413. One decrypted result exceeds the 1,000,000-byte response-page ceiling. No partial result is returned.

RATE_LIMITED

429. More than 600 requests in a 1-minute sliding window on the same API key. Slow down and retry after Retry-After. The response carries X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset. No credits are deducted. See Rate Limits.

CONCURRENCY_LIMIT

429. More than 50 concurrent requests were in flight on the same API key. Reduce concurrency and retry. No credits are deducted. The response carries X-Concurrency-Limit and X-Concurrency-Remaining, not X-RateLimit-*. See Rate Limits.

UPSTREAM_ERROR

502. The upstream platform returned an error, after our own automatic retries. Retry after a short backoff. Credits are refunded.

When error.details.retry_with is present, the same request will fail the same way: the response reads retryable: false, carries no Retry-After, and retry_with names the parameter to change (for example {"proxy": "auto"} on /v1/web/scrape). The same applies to a 504 that carries retry_with.

These error.details.reason values can accompany an UPSTREAM_ERROR (or, for url_unreachable, a RESOURCE_NOT_FOUND). All of them are refunded:

reasonWhereWhat it means
plain_fetch_failed/v1/web/scrape with proxy unset or basicA plain fetch of the page failed. retry_with.proxy is auto: send proxy=auto, which holds 5 credits and settles to actual use. Repeating the same parameters is unlikely to help.
url_unreachable/v1/web/scrape (404 RESOURCE_NOT_FOUND)The URL was rejected as invalid or does not resolve to a reachable site. Check the URL.
listings_feed_timeout/v1/facebook/group/posts, first pageThe group's listings feed (sort_by=CHRONOLOGICAL_LISTINGS) did not answer in time. retry_with.sort_by is CHRONOLOGICAL, which returns the group's posts in date order.
zero_results/v1/search/everywhereNo source returned rows and at least one could not be searched (or answered that it has nothing). sources_failed says why per source; sources_succeeded and coverage list the rest.
task_queuedSlow asynchronous data lanesThe provider queued the request but did not finish it in time. Retrying the same request within the hour picks up the queued job instead of starting a new one.

504. A GET call has 48 seconds to finish its upstream work; a few long-running endpoints, such as transcripts and Discover, have longer limits. When the 48 seconds run out, the call stops and answers 504 UPSTREAM_ERROR without Retry-After, and its credits are refunded; retry with backoff. If a call is still running at 54 seconds, it is answered with a 504 UPSTREAM_ERROR and Retry-After, its credits are refunded, and the work is left to finish. Retrying after Retry-After may then return it from the cache at no charge.

Every GET call is answered within 60 seconds, so a client timeout of 60 seconds never gives up before we answer. If your client gives up earlier, the call still finishes on our side and is charged, and the result is cached for the endpoint's cache window (freshness.cache_ttl_s on /v1/utility/endpoint). Sending the identical request again inside that window joins the call that is still running, or reads its result from the cache, at 0 credits. A request that sends Cache-Control: no-cache skips the cache and is charged again. latency_ms.recommended_timeout_s on the same endpoint is 1.5 times the measured p99 of successful uncached calls over the 7 days named in latency_ms.sampled, and it is null when fewer than 30 calls were measured.

To tell an upstream outage from a transient blip, check error.incident on the response, or GET /v1/status/{platform}. See Status on 5xx responses.

SERVICE_UNAVAILABLE

503. Several causes. An open circuit breaker or an upstream provider throttle carries Retry-After: 30. If the idempotency claim store is temporarily unavailable, a keyed request fails closed before billing with Retry-After: 1; retry with the same key. No balance lookup is attempted on that path, so credits_remaining is null and X-Credits-Remaining is omitted.

A 503 without a Retry-After header is not transient. It means the endpoint has been withdrawn because its upstream no longer serves it, and the message names the reason. Retrying will never succeed, so branch on the presence of Retry-After rather than on the status alone. Nothing is deducted in the first place on a withdrawn endpoint, since the router rejects it before billing, so there is no refund to wait for. Check the changelog for the replacement.

INTERNAL_ERROR

500. An unexpected error occurred on our side. Retry, and if it persists, contact support with the request_id. The server attempts the refund before responding. If it cannot commit, credits_used remains non-zero and the exact outstanding remainder is queued for automatic compensation rather than falsely reported as refunded.

RUN_NOT_FOUND

404. No /v1/prism/discover run with this run_id belongs to your account. A run started by any active key of your account is readable with any other; another account's run answers this code too, so a run id never confirms that someone else's run exists. details.reason is unknown_run. Start a run, or list your runs with GET /v1/prism/discover/runs. No credits are deducted.

RUN_STATE_CONFLICT

409. The run's state does not allow this call. details.reason names the state: run_completed (a completed run cannot be resumed, and a completed or partial run cannot be cancelled), run_failed or run_cancelled (it cannot be cancelled again), or run_not_completed (POST .../more needs a completed run; resume continues the others). details.next_step names the call that is valid in this state. No credits are deducted.

RUN_EXPIRED

410. The run's retention_days passed, so its rows were deleted. details.reason is retention_passed. Every run route answers this code for an expired run. A queued, running or paused run never expires while it is live. Start a new run. No credits are deducted.

NEEDS_ASYNC

422. A GET /v1/prism/discover request sent with async=false is over what one call can answer: more than 36 searches (keywords, platforms, lane kinds and pages, past-window pages included), more than 50 joins, comments read inline on more than 3 posts per group, or a limit over the call's cap. details.reason is one of legs, joins, comment_posts and limit, details.over lists every one, details.legs and details.joins give the counts against details.legs_max and details.joins_max, and details.causes names the request fields behind them, with details.estimate and a details.suggested_url without async=false, which runs the same request in the background and answers 202 with the run. No credits are deducted.

CURSOR_EXPIRED

410. A GET /v1/events cursor or a named consumer's position is older than the event log's 30-day retention, so events after it were deleted. details.reason is retention_passed. details.oldest_cursor is the position before the oldest event still kept, and details.missed_from and details.missed_until bound the window you missed (missed_from is null when the cursor carries no time). Restart from cursor=oldest (or details.oldest_cursor) and tell the user which window was missed. For a consumer the 410 commits nothing: its next read restarts at the oldest retained event and returns restarted_from once. See Events feed. No credits are deducted.

SCHEDULE_NOT_FOUND

404. No /v1/prism/discover schedule with this schedule_id belongs to your account. A schedule created with any active key of your account is readable with any other; another account's schedule, and one you deleted, answer this code too. details.reason is schedule_not_found. List your schedules with GET /v1/prism/discover/schedules. No credits are deducted.

Idempotency

Send an Idempotency-Key header to make a charging request safe to retry. The key is honored on every charging endpoint: the GET /v1/* catch-all and all six POST endpoints (/v1/prism/post-stats, /v1/prism/comment-lookup, /v1/prism/profiles, /v1/youtube/transcripts, /v1/youtube/videos, /v1/youtube/channels).

  • A retry with the same key and the same payload replays the stored terminal status and payload, charges nothing (X-Credits-Used: 0, X-Idempotent-Replay: true, Idempotent-Replayed: true), and does not re-run the upstream call. Replay billing metadata is refreshed: the current balance is returned when known.
  • The key's identity covers the request payload: query parameters for GET, and the canonical JSON body for POST. Reusing a key with a different payload returns IDEMPOTENCY_KEY_PAYLOAD_MISMATCH (422) rather than a false replay.
  • Keys are scoped per account. Two accounts can use the same opaque key independently.
  • A same-key GET that arrives while the original is still running waits up to 20 seconds for it and replays its result. A request still running after that, or a same-key POST, returns IDEMPOTENCY_IN_PROGRESS (409) with a Retry-After of 1 to 30 seconds that grows with the original's running time. It is not dispatched or charged; retry with the same key.
  • An unfinished claim expires after 5 minutes so a crashed request cannot lock the key indefinitely.
  • A completed outcome whose body exceeds 64 KB is stored compressed; one that is still over 160 KB compressed returns IDEMPOTENCY_REPLAY_UNAVAILABLE (409). The 64 KB is measured on the JSON response body. A new key sends the request again, which is served from the cache at 0 credits while the endpoint's cache window is open and is billed after it.
  • If the claim store is unavailable, the keyed request fails closed at 503 with Retry-After: 1 before billing.
  • Completed outcomes are retained for 24 hours. Use a fresh UUIDv4 per logical operation.
  • GET SSE cannot be combined with Idempotency-Key; the unsupported combination returns an unbilled 400 INVALID_REQUEST before dispatch. POST batch streams keep sync-equivalent replay.

Refund rules

Credits are refunded automatically when:

  • Upstream returns a 5xx error (UPSTREAM_ERROR, 502 or 504).
  • The circuit breaker rejects the request, or the upstream provider rate-limits us (SERVICE_UNAVAILABLE, 503).
  • An unexpected server error occurs (INTERNAL_ERROR, 500).
  • Upstream returns 200 with an empty body, for example a nonexistent handle, triggering RESOURCE_NOT_FOUND (404).

Credits are never deducted for these outcomes, so no refund is needed:

  • Cache hits (X-Cache: HIT, X-Credits-Used: 0).
  • METHOD_NOT_ALLOWED (405), IDEMPOTENCY_KEY_CONFLICT (409), IDEMPOTENCY_IN_PROGRESS (409), IDEMPOTENCY_REPLAY_UNAVAILABLE (409), and IDEMPOTENCY_KEY_PAYLOAD_MISMATCH (422).
  • KEY_BUDGET_EXCEEDED (402). The request is refused before the charge lands, and the key's own usage counter is left where it was.
  • Idempotent replays. The original charge already appears on your earlier request.

Client errors (MISSING_API_KEY, INVALID_API_KEY, INVALID_REQUEST, ENDPOINT_NOT_FOUND) are rejected before any deduction. The one exception is an INVALID_REQUEST that only the data source can detect, such as a page past the end of a search: that call is charged and then refunded in full before the response is sent. See Credits for the full refund matrix.

Handling retries

Retry only the errors marked retryable above: IDEMPOTENCY_IN_PROGRESS (409), RATE_LIMITED or CONCURRENCY_LIMIT (429), and transient 500, 502, 503, or 504 responses. Honor Retry-After: idempotency contention sends 1 to 30 (it grows with the original request's running time), claim-store outages send 1, circuit/provider 503s send 30, rate-limit 429s send the seconds until reset, and concurrency 429s send a short hint.

Otherwise back off exponentially with full jitter, so a pool of your own workers does not resynchronize and stampede the cap together. Cap the attempts and surface the error rather than retrying forever. Never retry a 4xx client error other than 429: the request is deterministic and will fail the same way.

Retryable failures attempt an immediate refund; if that database mutation fails, exact compensation is queued and the response keeps a non-zero committed credits_used. Add an Idempotency-Key header (see Idempotency) so a network-level retry of a successful sync call replays rather than re-runs.

Each loop below retries only the transient statuses (429, 500, 502, 503, 504), honors Retry-After when present, falls back to exponential backoff with jitter, and gives up after five attempts.

Terminal
#!/usr/bin/env bash
url="https://www.socialcrawl.dev/v1/tiktok/profile?handle=charlidamelio"
max_retries=5
attempt=0

while :; do
  body=$(mktemp); headers=$(mktemp)
  status=$(curl -s -o "$body" -D "$headers" -w '%{http_code}' \
    -H "x-api-key: $SOCIALCRAWL_API_KEY" "$url")

  if [ "$status" -lt 400 ]; then
    cat "$body"; rm -f "$body" "$headers"; break
  fi

  case "$status" in
    429|500|502|503|504) ;;  # retryable, fall through
    *) echo "Non-retryable $status"; cat "$body"; rm -f "$body" "$headers"; exit 1 ;;
  esac

  attempt=$((attempt + 1))
  if [ "$attempt" -gt "$max_retries" ]; then
    echo "Gave up after $max_retries retries (last status $status)"
    rm -f "$body" "$headers"; exit 1
  fi

  # Retry-After (seconds) wins; else exponential backoff 2^attempt with jitter.
  retry_after=$(grep -i '^retry-after:' "$headers" | tr -d '\r' | awk '{print $2}')
  if [ -n "$retry_after" ]; then
    delay="$retry_after"
  else
    delay=$(awk "BEGIN{srand(); print (2 ^ $attempt) * (0.5 + rand() / 2)}")
  fi
  rm -f "$body" "$headers"
  sleep "$delay"
done
Python
import os
import random
import time

import requests  # pip install requests

BASE_URL = "https://www.socialcrawl.dev/v1"
RETRYABLE = {429, 500, 502, 503, 504}


def get_with_retry(path, params=None, max_retries=5):
    headers = {"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]}
    for attempt in range(max_retries + 1):
        resp = requests.get(f"{BASE_URL}{path}", params=params, headers=headers)
        # Success, or a client error we must not retry: hand it back / raise.
        if resp.ok or resp.status_code not in RETRYABLE or attempt == max_retries:
            resp.raise_for_status()
            return resp.json()
        # Retry-After (seconds) wins; else exponential backoff with full jitter.
        retry_after = resp.headers.get("Retry-After")
        delay = float(retry_after) if retry_after else (2 ** attempt) * (0.5 + random.random() / 2)
        time.sleep(delay)


data = get_with_retry("/tiktok/profile", {"handle": "charlidamelio"})
print(data)
JavaScript
// Node 18+ (built-in fetch). No dependencies.
const BASE_URL = "https://www.socialcrawl.dev/v1";
const RETRYABLE = new Set([429, 500, 502, 503, 504]);
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

async function getWithRetry(path, { params = {}, maxRetries = 5 } = {}) {
  const url = new URL(BASE_URL + path);
  for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
  const headers = { "x-api-key": process.env.SOCIALCRAWL_API_KEY };

  for (let attempt = 0; ; attempt++) {
    const res = await fetch(url, { headers });
    if (res.ok || !RETRYABLE.has(res.status)) {
      if (!res.ok)
        throw new Error(`SocialCrawl ${res.status}: ${await res.text()}`);
      return res.json();
    }
    if (attempt >= maxRetries) {
      throw new Error(
        `Gave up after ${maxRetries} retries (last status ${res.status})`,
      );
    }
    // Retry-After (seconds) wins; else exponential backoff with full jitter.
    const retryAfter = Number(res.headers.get("Retry-After"));
    const delayMs =
      Number.isFinite(retryAfter) && retryAfter > 0
        ? retryAfter * 1000
        : 2 ** attempt * 1000 * (0.5 + Math.random() / 2);
    await sleep(delayMs);
  }
}

const data = await getWithRetry("/tiktok/profile", {
  params: { handle: "charlidamelio" },
});
console.log(data);

Status on 5xx responses

Every 500, 502, 503 and 504 body carries two additive fields inside error:

  • error.status_url: the public status page, filtered to the platform when one is known, for example https://www.socialcrawl.dev/status?platform=youtube.
  • error.incident: present only when an open public incident explains the failure. It has id, title, status, impact and url.
Response
{
  "status_url": "https://www.socialcrawl.dev/status?platform=youtube",
  "incident": {
    "id": "k3v9w2m8q1zt",
    "title": "Elevated request failures on YouTube",
    "status": "investigating",
    "impact": "major",
    "url": "https://www.socialcrawl.dev/status/incidents/k3v9w2m8q1zt"
  }
}

When error.incident is present, retrying sooner will not help: wait for the incident to clear, then resume. Without it, treat the failure as transient and honour Retry-After. Status has the full recipe in cURL, TypeScript and Python, and explains how the status page decides each state.

Debugging tips

  • Every error envelope carries a doc_url pointing at the matching section on this page, for example https://www.socialcrawl.dev/docs/errors#insufficient-credits.
  • request_id matches the X-Request-Id header and the request_id column on Dashboard → Activity Logs.
  • Persistent 502s on a known-good input usually indicate an upstream outage. Check error.incident on the response, the status page, or GET /v1/status/{platform}.
  • A high retry count on a slow-but-successful call is surfaced in the X-Upstream-Retries response header.

Next steps