# Errors and retries (/docs/handle-errors) Which failures to retry, how to back off, and the loop to copy in cURL, Python and Node. A failed call returns the same JSON envelope every time, and `error.retryable` says whether sending it again can succeed. This page is the retry policy and the loop to copy. [Errors](/docs/errors.md) is the full list of codes, with what each one costs. ## What to retry [#what-to-retry] Retry only the errors marked retryable on the [errors page](/docs/errors.md#which-errors-are-safe-to-retry): `IDEMPOTENCY_IN_PROGRESS` (409), `RATE_LIMITED` or `CONCURRENCY_LIMIT` (429), and transient `500`, `502`, `503`, or `504` responses. Honor `Retry-After`: idempotency contention sends 1 to 30 (it grows with the original request's running time), claim-store outages send `1`, other transient 503s send a wait that depends on the cause, rate-limit 429s send the seconds until reset, and concurrency 429s send a short hint. Always read the header rather than assume a fixed wait. Never retry a `4xx` client error other than `429`: the request is deterministic and will fail the same way. That includes invalid authentication, insufficient credits, a key budget breach, invalid parameters and an idempotency payload mismatch. A withdrawn endpoint returns `503 SERVICE_UNAVAILABLE` permanently without `Retry-After`; surface that response instead of retrying it. ## Back off with jitter [#back-off-with-jitter] Back off exponentially with full jitter, so a pool of your own workers does not resynchronize and stampede the cap together. Cap the attempts and surface the error rather than retrying forever. Each loop below retries only the transient statuses (429, 500, 502, 503, 504), honors `Retry-After` when present, falls back to exponential backoff with jitter, and gives up after five attempts. **cURL** ```bash title="Terminal" #!/usr/bin/env bash url="https://www.socialcrawl.dev/v1/tiktok/profile?handle=charlidamelio" max_retries=5 attempt=0 while :; do body=$(mktemp); headers=$(mktemp) status=$(curl -s -o "$body" -D "$headers" -w '%{http_code}' \ -H "x-api-key: $SOCIALCRAWL_API_KEY" "$url") if [ "$status" -lt 400 ]; then cat "$body"; rm -f "$body" "$headers"; break fi case "$status" in 429|500|502|503|504) ;; # retryable, fall through *) echo "Non-retryable $status"; cat "$body"; rm -f "$body" "$headers"; exit 1 ;; esac attempt=$((attempt + 1)) if [ "$attempt" -gt "$max_retries" ]; then echo "Gave up after $max_retries retries (last status $status)" rm -f "$body" "$headers"; exit 1 fi # Retry-After (seconds) wins; else exponential backoff 2^attempt with jitter. retry_after=$(grep -i '^retry-after:' "$headers" | tr -d '\r' | awk '{print $2}') if [ -n "$retry_after" ]; then delay="$retry_after" else delay=$(awk "BEGIN{srand(); print (2 ^ $attempt) * (0.5 + rand() / 2)}") fi rm -f "$body" "$headers" sleep "$delay" done ``` **Python** ```python title="Python" import os import random import time import requests # pip install requests BASE_URL = "https://www.socialcrawl.dev/v1" RETRYABLE = {429, 500, 502, 503, 504} def get_with_retry(path, params=None, max_retries=5): headers = {"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]} for attempt in range(max_retries + 1): resp = requests.get(f"{BASE_URL}{path}", params=params, headers=headers) # Success, or a client error we must not retry: hand it back / raise. if resp.ok or resp.status_code not in RETRYABLE or attempt == max_retries: resp.raise_for_status() return resp.json() # Retry-After (seconds) wins; else exponential backoff with full jitter. retry_after = resp.headers.get("Retry-After") delay = float(retry_after) if retry_after else (2 ** attempt) * (0.5 + random.random() / 2) time.sleep(delay) data = get_with_retry("/tiktok/profile", {"handle": "charlidamelio"}) print(data) ``` **Node** ```js title="JavaScript" // Node 18+ (built-in fetch). No dependencies. const BASE_URL = "https://www.socialcrawl.dev/v1"; const RETRYABLE = new Set([429, 500, 502, 503, 504]); const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); async function getWithRetry(path, { params = {}, maxRetries = 5 } = {}) { const url = new URL(BASE_URL + path); for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v); const headers = { "x-api-key": process.env.SOCIALCRAWL_API_KEY }; for (let attempt = 0; ; attempt++) { const res = await fetch(url, { headers }); if (res.ok || !RETRYABLE.has(res.status)) { if (!res.ok) throw new Error(`SocialCrawl ${res.status}: ${await res.text()}`); return res.json(); } if (attempt >= maxRetries) { throw new Error( `Gave up after ${maxRetries} retries (last status ${res.status})`, ); } // Retry-After (seconds) wins; else exponential backoff with full jitter. const retryAfter = Number(res.headers.get("Retry-After")); const delayMs = Number.isFinite(retryAfter) && retryAfter > 0 ? retryAfter * 1000 : 2 ** attempt * 1000 * (0.5 + Math.random() / 2); await sleep(delayMs); } } const data = await getWithRetry("/tiktok/profile", { params: { handle: "charlidamelio" }, }); console.log(data); ``` ## Retry safely [#retry-safely] Retryable failures attempt an immediate refund; if that database mutation fails, exact compensation is queued and the response keeps a non-zero committed `credits_used`. Add an `Idempotency-Key` header so a network-level retry of a successful sync call replays rather than re-runs. [Retry safely](/docs/idempotency.md) lists which calls take a key and which ignore it. ## Read the failure [#read-the-failure] Branch on these fields of the failed response, in this order. - **`error.type`** is the stable code to branch on. - **`error.retryable`** is `true` when the identical request can succeed, so you do not need to keep your own list of codes. - **`credits_used`** is the net committed charge. A non-zero value on a retryable failure means the immediate refund did not commit and automatic compensation is pending. - **`request_id`** matches the `X-Request-Id` header. - **`error.doc_url`** points at the entry for that code on the errors page. ```json title="Response" { "success": false, "error": { "type": "RATE_LIMITED", "status": 429, "retryable": true, "doc_url": "https://www.socialcrawl.dev/docs/errors#rate-limited" }, "credits_used": 0, "request_id": "req-abc123" } ``` The block is trimmed to the fields above. A real response also carries `error.message` and `error.details`. ## Log what support needs [#log-what-support-needs] Log `request_id` and the `X-Request-Id` header for every failure. A high retry count on a slow-but-successful call is surfaced in the `X-Upstream-Retries` response header, so log that too. A `500`, `502`, `503` or `504` body also carries `error.status_url`, and `error.incident` when an open incident explains it. [Is it down, or is it me?](/docs/status.md) covers what to do with them. ## Ask for help [#ask-for-help] Send the `request_id` of the failing call to [socialcrawl@ridio.io](mailto:socialcrawl@ridio.io), or read [Support](/docs/support.md) first. A request id lets us trace the exact call. ## Next steps - [Errors](/docs/errors.md): Every error code, its status, whether to retry it, and what it costs. - [Retry safely](/docs/idempotency.md): Send an Idempotency-Key so a retried call returns the first answer for 0 credits. - [Production checklist](/docs/production-checklist.md): Timeouts, retries, paging and spend controls before you go live.