Retry safely with Idempotency-Key

A timeout leaves you not knowing whether a call ran. Put a unique value in an Idempotency-Key header, and send the retry with the same value: SocialCrawl returns the first call's answer for 0 credits instead of running it, and billing it, again.

KEY="$(uuidgen)"

# The first call runs and is billed.
curl "https://www.socialcrawl.dev/v1/tiktok/profile?handle=nasa" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Idempotency-Key: $KEY"

# The retry, with the same key, replays that answer and costs 0 credits.
curl -i "https://www.socialcrawl.dev/v1/tiktok/profile?handle=nasa" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Idempotency-Key: $KEY"

The second call did not reach TikTok. It came back with the first call's body, credits_used: 0 and idempotent_replay: true, and it was not billed. Use one key per logical request, keep it across that request's retries, and make a new one for the next request.

What does a replay look like?

The status and the body are the first call's, with the billing fields refreshed:

{
  "success": true,
  "platform": "tiktok",
  "endpoint": "/v1/tiktok/profile",
  "data": {
    "author": { "username": "nasa", "followers": 3900000 }
  },
  "credits_used": 0,
  "credits_remaining": 99,
  "request_id": "req-abc123",
  "idempotent_replay": true
}

Which calls take a key?

A key covers the method, the path, the query and the JSON body. Key order inside the body does not matter, and array order does: the same members in a different order are a different request. Send the same key with a different payload and you get 422 IDEMPOTENCY_KEY_PAYLOAD_MISMATCH, never a false replay. On the GET and batch calls a key belongs to your account, so every API key on the account shares it; on the job submits, the webhook endpoint calls and the cohort calls it belongs to the one API key that sent it.

Which calls ignore the key?

These calls run every time you send them, key or not. A retry after a timeout can start a second browser session, a second monitor or a second upload, and bill it.

• POST /v1/web/sessions
• POST /v1/web/sessions/{session_id}/execute
• POST /v1/web/agent
• POST /v1/web/monitors
• POST /v1/web/parse
• POST /v1/trustpilot/reviews/history
• POST /v1/prism/discover/inputs
• POST /v1/monitors

Before you resend one of these after a timeout, list what exists (GET /v1/web/sessions, GET /v1/web/monitors, GET /v1/monitors) and send it again only if the first call did not land. PATCH and DELETE calls are safe to repeat without a key, because sending them twice leaves the same state.

What happens while the first call is still running?

A GET retry with the same key waits up to 20 seconds for the first call to finish, then replays its answer. If the first call is still running after that, or the retry is a POST, you get 409 IDEMPOTENCY_IN_PROGRESS with a Retry-After of 1 to 30 seconds, which grows with how long the first call has run. Nothing was sent or billed: wait Retry-After, then send the same key and payload again.

When is a call not replayed?

• The first call failed with a retryable error. It is not stored, so the retry with the same key runs again, which is what you want. Only a finished answer is stored and replayed.
• The answer was too large to keep. An answer over 64 KB of JSON is stored compressed. One that is still over 160 KB compressed cannot be replayed: the retry gets 409 IDEMPOTENCY_REPLAY_UNAVAILABLE and is not charged. Send the request with a new key. While the endpoint's cache window is open it is served from the cache for 0 credits.
• The key is older than its window. 24 hours for the stored answers, 7 days for the job submits. After that the same key is a new request.
• It was a dry run. A GET with dry_run=1 stores nothing, so the same key still works for the real call.
• It was a streamed GET. A GET that asks for a stream (Accept: text/event-stream, or the endpoint's own stream parameter) and carries an Idempotency-Key is a free 400 INVALID_REQUEST, because an interrupted stream has no whole answer to replay. Ask for JSON when you need a replay. See Streaming.
• The key store was unavailable. You get 503 SERVICE_UNAVAILABLE with Retry-After: 1 before anything runs. Retry with the same key.

Errors you can get

None of these is charged. The full catalogue is in Errors.

A retry loop that keys every call

const KEY = process.env.SOCIALCRAWL_API_KEY!;
const RETRYABLE = new Set([
  "IDEMPOTENCY_IN_PROGRESS",
  "RATE_LIMITED",
  "CONCURRENCY_LIMIT",
  "UPSTREAM_ERROR",
  "SERVICE_UNAVAILABLE",
  "INTERNAL_ERROR",
]);

async function getWithRetry(url: string, attempts = 4) {
  // One key for this request, reused on every retry of it.
  const idempotencyKey = crypto.randomUUID();
  for (let attempt = 1; ; attempt++) {
    try {
      const res = await fetch(url, {
        headers: { "x-api-key": KEY, "Idempotency-Key": idempotencyKey },
        signal: AbortSignal.timeout(30_000),
      });
      const body = await res.json();
      if (body.success || !RETRYABLE.has(body.error?.type) || attempt === attempts) {
        return body;
      }
      const wait = Number(res.headers.get("Retry-After") ?? 2 ** attempt);
      await new Promise((resolve) => setTimeout(resolve, wait * 1000));
    } catch (error) {
      // A timeout or a dropped connection: the call may have run, so retry with the same key.
      if (attempt === attempts) throw error;
    }
  }
}

const body = await getWithRetry("https://www.socialcrawl.dev/v1/tiktok/profile?handle=nasa");
console.log(body.idempotent_replay ?? false, body.credits_used);

Every retry of one call reuses its key, so a timeout that hid a finished call costs nothing the second time. The Production checklist has the full wrapper with per-endpoint timeouts and logging.

Webhook deliveries you receive

Idempotency-Key is for calls you send. Deliveries SocialCrawl sends you can also arrive twice, because every channel retries until your endpoint answers 2xx. De-duplicate those on the id each channel names: see Webhooks and events.

Next steps