SocialCrawl

Handle errors and retries

Which failures to retry, how to back off, and the loop to copy in cURL, Python and Node.

A failed call returns the same JSON envelope every time, and error.retryable says whether sending it again can succeed. This page is the retry policy and the loop to copy. Errors is the full list of codes, with what each one costs.

What to retry

Retry only the errors marked retryable on the errors page: IDEMPOTENCY_IN_PROGRESS (409), RATE_LIMITED or CONCURRENCY_LIMIT (429), and transient 500, 502, 503, or 504 responses. Honor Retry-After: idempotency contention sends 1 to 30 (it grows with the original request's running time), claim-store outages send 1, other transient 503s send a wait that depends on the cause, rate-limit 429s send the seconds until reset, and concurrency 429s send a short hint. Always read the header rather than assume a fixed wait.

Never retry a 4xx client error other than 429: the request is deterministic and will fail the same way. That includes invalid authentication, insufficient credits, a key budget breach, invalid parameters and an idempotency payload mismatch. A withdrawn endpoint returns 503 SERVICE_UNAVAILABLE permanently without Retry-After; surface that response instead of retrying it.

Back off with jitter

Back off exponentially with full jitter, so a pool of your own workers does not resynchronize and stampede the cap together. Cap the attempts and surface the error rather than retrying forever.

Each loop below retries only the transient statuses (429, 500, 502, 503, 504), honors Retry-After when present, falls back to exponential backoff with jitter, and gives up after five attempts.

Terminal
#!/usr/bin/env bash
url="https://www.socialcrawl.dev/v1/tiktok/profile?handle=charlidamelio"
max_retries=5
attempt=0

while :; do
  body=$(mktemp); headers=$(mktemp)
  status=$(curl -s -o "$body" -D "$headers" -w '%{http_code}' \
    -H "x-api-key: $SOCIALCRAWL_API_KEY" "$url")

  if [ "$status" -lt 400 ]; then
    cat "$body"; rm -f "$body" "$headers"; break
  fi

  case "$status" in
    429|500|502|503|504) ;;  # retryable, fall through
    *) echo "Non-retryable $status"; cat "$body"; rm -f "$body" "$headers"; exit 1 ;;
  esac

  attempt=$((attempt + 1))
  if [ "$attempt" -gt "$max_retries" ]; then
    echo "Gave up after $max_retries retries (last status $status)"
    rm -f "$body" "$headers"; exit 1
  fi

  # Retry-After (seconds) wins; else exponential backoff 2^attempt with jitter.
  retry_after=$(grep -i '^retry-after:' "$headers" | tr -d '\r' | awk '{print $2}')
  if [ -n "$retry_after" ]; then
    delay="$retry_after"
  else
    delay=$(awk "BEGIN{srand(); print (2 ^ $attempt) * (0.5 + rand() / 2)}")
  fi
  rm -f "$body" "$headers"
  sleep "$delay"
done
Python
import os
import random
import time

import requests  # pip install requests

BASE_URL = "https://www.socialcrawl.dev/v1"
RETRYABLE = {429, 500, 502, 503, 504}


def get_with_retry(path, params=None, max_retries=5):
    headers = {"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]}
    for attempt in range(max_retries + 1):
        resp = requests.get(f"{BASE_URL}{path}", params=params, headers=headers)
        # Success, or a client error we must not retry: hand it back / raise.
        if resp.ok or resp.status_code not in RETRYABLE or attempt == max_retries:
            resp.raise_for_status()
            return resp.json()
        # Retry-After (seconds) wins; else exponential backoff with full jitter.
        retry_after = resp.headers.get("Retry-After")
        delay = float(retry_after) if retry_after else (2 ** attempt) * (0.5 + random.random() / 2)
        time.sleep(delay)


data = get_with_retry("/tiktok/profile", {"handle": "charlidamelio"})
print(data)
JavaScript
// Node 18+ (built-in fetch). No dependencies.
const BASE_URL = "https://www.socialcrawl.dev/v1";
const RETRYABLE = new Set([429, 500, 502, 503, 504]);
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

async function getWithRetry(path, { params = {}, maxRetries = 5 } = {}) {
  const url = new URL(BASE_URL + path);
  for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
  const headers = { "x-api-key": process.env.SOCIALCRAWL_API_KEY };

  for (let attempt = 0; ; attempt++) {
    const res = await fetch(url, { headers });
    if (res.ok || !RETRYABLE.has(res.status)) {
      if (!res.ok)
        throw new Error(`SocialCrawl ${res.status}: ${await res.text()}`);
      return res.json();
    }
    if (attempt >= maxRetries) {
      throw new Error(
        `Gave up after ${maxRetries} retries (last status ${res.status})`,
      );
    }
    // Retry-After (seconds) wins; else exponential backoff with full jitter.
    const retryAfter = Number(res.headers.get("Retry-After"));
    const delayMs =
      Number.isFinite(retryAfter) && retryAfter > 0
        ? retryAfter * 1000
        : 2 ** attempt * 1000 * (0.5 + Math.random() / 2);
    await sleep(delayMs);
  }
}

const data = await getWithRetry("/tiktok/profile", {
  params: { handle: "charlidamelio" },
});
console.log(data);

Retry safely

Retryable failures attempt an immediate refund; if that database mutation fails, exact compensation is queued and the response keeps a non-zero committed credits_used. Add an Idempotency-Key header so a network-level retry of a successful sync call replays rather than re-runs. Retry safely lists which calls take a key and which ignore it.

Read the failure

Branch on these fields of the failed response, in this order.

  • error.type is the stable code to branch on.
  • error.retryable is true when the identical request can succeed, so you do not need to keep your own list of codes.
  • credits_used is the net committed charge. A non-zero value on a retryable failure means the immediate refund did not commit and automatic compensation is pending.
  • request_id matches the X-Request-Id header.
  • error.doc_url points at the entry for that code on the errors page.
Response
{
  "success": false,
  "error": {
    "type": "RATE_LIMITED",
    "status": 429,
    "retryable": true,
    "doc_url": "https://www.socialcrawl.dev/docs/errors#rate-limited"
  },
  "credits_used": 0,
  "request_id": "req-abc123"
}

The block is trimmed to the fields above. A real response also carries error.message and error.details.

Log what support needs

Log request_id and the X-Request-Id header for every failure. A high retry count on a slow-but-successful call is surfaced in the X-Upstream-Retries response header, so log that too.

A 500, 502, 503 or 504 body also carries error.status_url, and error.incident when an open incident explains it. Is it down, or is it me? covers what to do with them.

Ask for help

Send the request_id of the failing call to socialcrawl@ridio.io, or read Support first. A request id lets us trace the exact call.

Next steps

On this page