Handle errors and retries
Which failures to retry, how to back off, and the loop to copy in cURL, Python and Node.
A failed call returns the same JSON envelope every time, and error.retryable says whether sending it again can succeed. This page is the retry policy and the loop to copy. Errors is the full list of codes, with what each one costs.
What to retry
Retry only the errors marked retryable on the errors page: IDEMPOTENCY_IN_PROGRESS (409), RATE_LIMITED or CONCURRENCY_LIMIT (429), and transient 500, 502, 503, or 504 responses. Honor Retry-After: idempotency contention sends 1 to 30 (it grows with the original request's running time), claim-store outages send 1, other transient 503s send a wait that depends on the cause, rate-limit 429s send the seconds until reset, and concurrency 429s send a short hint. Always read the header rather than assume a fixed wait.
Never retry a 4xx client error other than 429: the request is deterministic and will fail the same way. That includes invalid authentication, insufficient credits, a key budget breach, invalid parameters and an idempotency payload mismatch. A withdrawn endpoint returns 503 SERVICE_UNAVAILABLE permanently without Retry-After; surface that response instead of retrying it.
Back off with jitter
Back off exponentially with full jitter, so a pool of your own workers does not resynchronize and stampede the cap together. Cap the attempts and surface the error rather than retrying forever.
Each loop below retries only the transient statuses (429, 500, 502, 503, 504), honors Retry-After when present, falls back to exponential backoff with jitter, and gives up after five attempts.
#!/usr/bin/env bash
url="https://www.socialcrawl.dev/v1/tiktok/profile?handle=charlidamelio"
max_retries=5
attempt=0
while :; do
body=$(mktemp); headers=$(mktemp)
status=$(curl -s -o "$body" -D "$headers" -w '%{http_code}' \
-H "x-api-key: $SOCIALCRAWL_API_KEY" "$url")
if [ "$status" -lt 400 ]; then
cat "$body"; rm -f "$body" "$headers"; break
fi
case "$status" in
429|500|502|503|504) ;; # retryable, fall through
*) echo "Non-retryable $status"; cat "$body"; rm -f "$body" "$headers"; exit 1 ;;
esac
attempt=$((attempt + 1))
if [ "$attempt" -gt "$max_retries" ]; then
echo "Gave up after $max_retries retries (last status $status)"
rm -f "$body" "$headers"; exit 1
fi
# Retry-After (seconds) wins; else exponential backoff 2^attempt with jitter.
retry_after=$(grep -i '^retry-after:' "$headers" | tr -d '\r' | awk '{print $2}')
if [ -n "$retry_after" ]; then
delay="$retry_after"
else
delay=$(awk "BEGIN{srand(); print (2 ^ $attempt) * (0.5 + rand() / 2)}")
fi
rm -f "$body" "$headers"
sleep "$delay"
doneimport os
import random
import time
import requests # pip install requests
BASE_URL = "https://www.socialcrawl.dev/v1"
RETRYABLE = {429, 500, 502, 503, 504}
def get_with_retry(path, params=None, max_retries=5):
headers = {"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]}
for attempt in range(max_retries + 1):
resp = requests.get(f"{BASE_URL}{path}", params=params, headers=headers)
# Success, or a client error we must not retry: hand it back / raise.
if resp.ok or resp.status_code not in RETRYABLE or attempt == max_retries:
resp.raise_for_status()
return resp.json()
# Retry-After (seconds) wins; else exponential backoff with full jitter.
retry_after = resp.headers.get("Retry-After")
delay = float(retry_after) if retry_after else (2 ** attempt) * (0.5 + random.random() / 2)
time.sleep(delay)
data = get_with_retry("/tiktok/profile", {"handle": "charlidamelio"})
print(data)// Node 18+ (built-in fetch). No dependencies.
const BASE_URL = "https://www.socialcrawl.dev/v1";
const RETRYABLE = new Set([429, 500, 502, 503, 504]);
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
async function getWithRetry(path, { params = {}, maxRetries = 5 } = {}) {
const url = new URL(BASE_URL + path);
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
const headers = { "x-api-key": process.env.SOCIALCRAWL_API_KEY };
for (let attempt = 0; ; attempt++) {
const res = await fetch(url, { headers });
if (res.ok || !RETRYABLE.has(res.status)) {
if (!res.ok)
throw new Error(`SocialCrawl ${res.status}: ${await res.text()}`);
return res.json();
}
if (attempt >= maxRetries) {
throw new Error(
`Gave up after ${maxRetries} retries (last status ${res.status})`,
);
}
// Retry-After (seconds) wins; else exponential backoff with full jitter.
const retryAfter = Number(res.headers.get("Retry-After"));
const delayMs =
Number.isFinite(retryAfter) && retryAfter > 0
? retryAfter * 1000
: 2 ** attempt * 1000 * (0.5 + Math.random() / 2);
await sleep(delayMs);
}
}
const data = await getWithRetry("/tiktok/profile", {
params: { handle: "charlidamelio" },
});
console.log(data);Retry safely
Retryable failures attempt an immediate refund; if that database mutation fails, exact compensation is queued and the response keeps a non-zero committed credits_used. Add an Idempotency-Key header so a network-level retry of a successful sync call replays rather than re-runs. Retry safely lists which calls take a key and which ignore it.
Read the failure
Branch on these fields of the failed response, in this order.
error.typeis the stable code to branch on.error.retryableistruewhen the identical request can succeed, so you do not need to keep your own list of codes.credits_usedis the net committed charge. A non-zero value on a retryable failure means the immediate refund did not commit and automatic compensation is pending.request_idmatches theX-Request-Idheader.error.doc_urlpoints at the entry for that code on the errors page.
{
"success": false,
"error": {
"type": "RATE_LIMITED",
"status": 429,
"retryable": true,
"doc_url": "https://www.socialcrawl.dev/docs/errors#rate-limited"
},
"credits_used": 0,
"request_id": "req-abc123"
}The block is trimmed to the fields above. A real response also carries error.message and error.details.
Log what support needs
Log request_id and the X-Request-Id header for every failure. A high retry count on a slow-but-successful call is surfaced in the X-Upstream-Retries response header, so log that too.
A 500, 502, 503 or 504 body also carries error.status_url, and error.incident when an open incident explains it. Is it down, or is it me? covers what to do with them.
Ask for help
Send the request_id of the failing call to socialcrawl@ridio.io, or read Support first. A request id lets us trace the exact call.
