# Production Checklist (/docs/production-checklist)

Timeouts, retries, idempotency, pagination, caching, key isolation, billing controls, and logging before release

Use one server-side request wrapper for authentication, timeouts, retries, and request tracing. Keep endpoint policy outside the wrapper so each call can use its own timeout and retry budget.

## Start with one request wrapper

```typescript title="TypeScript"
const BASE_URL = "https://www.socialcrawl.dev";
const BASE_BACKOFF_MS = 500;
const MAX_BACKOFF_MS = 8_000;

const TRANSIENT_ERROR_TYPES = new Set([
  "IDEMPOTENCY_IN_PROGRESS",
  "RATE_LIMITED",
  "CONCURRENCY_LIMIT",
  "UPSTREAM_ERROR",
  "SERVICE_UNAVAILABLE",
  "INTERNAL_ERROR",
]);
const TRANSIENT_HTTP_STATUSES = new Set([429, 500, 502, 504]);

interface SocialCrawlEnvelope<T> {
  success: boolean;
  data?: T;
  error?: {
    type: string;
    message: string;
    status: number;
  };
  request_id: string;
  credits_used: number;
  credits_remaining: number | null;
  cached?: boolean;
  idempotent_replay?: true;
}

interface RequestOptions {
  endpoint: `/v1/${string}`;
  query?: Record<string, string>;
  attemptTimeoutMs: number;
  maxRetries?: number;
  onAttempt?: (event: AttemptLog) => void;
}

interface AttemptDetails {
  status: number | null;
  requestId: string | null;
  errorType: string | null;
  creditsUsed: number | null;
  cacheState: string | null;
}

interface AttemptLog extends AttemptDetails {
  endpoint: string;
  attempt: number;
  latencyMs: number;
}

interface RequestFailure {
  endpoint: string;
  status: number | null;
  errorType: string;
  requestId: string | null;
  creditsUsed: number | null;
  cacheState: string | null;
  attemptCount: number;
}

class SocialCrawlRequestError extends Error {
  readonly details: RequestFailure;

  constructor(message: string, details: RequestFailure) {
    super(message);
    this.name = "SocialCrawlRequestError";
    this.details = details;
  }
}

function isRecord(value: unknown): value is Record<string, unknown> {
  return typeof value === "object" && value !== null;
}

function isEnvelope(value: unknown): value is SocialCrawlEnvelope<unknown> {
  return (
    isRecord(value) &&
    typeof value.success === "boolean" &&
    typeof value.request_id === "string" &&
    typeof value.credits_used === "number"
  );
}

function parseRetryAfter(value: string | null): number | undefined {
  if (value === null) return undefined;
  const seconds = Number(value);
  return Number.isFinite(seconds) && seconds >= 0 ? seconds * 1_000 : undefined;
}

function isRetryable(
  errorType: string,
  retryAfterMs: number | undefined,
): boolean {
  if (!TRANSIENT_ERROR_TYPES.has(errorType)) return false;

  // A withdrawn endpoint returns this type without Retry-After and is permanent.
  if (errorType === "SERVICE_UNAVAILABLE" && retryAfterMs === undefined) {
    return false;
  }

  return true;
}

function isRetryableHttpFailure(
  status: number,
  retryAfterMs: number | undefined,
): boolean {
  return (
    TRANSIENT_HTTP_STATUSES.has(status) ||
    (status === 503 && retryAfterMs !== undefined)
  );
}

const sleep = (milliseconds: number) =>
  new Promise<void>((resolve) => setTimeout(resolve, milliseconds));

function backoffMs(attempt: number, retryAfterMs = 0): number {
  const exponentialCap = Math.min(
    MAX_BACKOFF_MS,
    BASE_BACKOFF_MS * 2 ** attempt,
  );
  return Math.max(retryAfterMs, Math.random() * exponentialCap);
}

async function requestSocialCrawl<T>({
  endpoint,
  query = {},
  attemptTimeoutMs,
  maxRetries = 3,
  onAttempt,
}: RequestOptions): Promise<SocialCrawlEnvelope<T>> {
  const apiKey = process.env.SOCIALCRAWL_API_KEY;
  if (!apiKey) throw new Error("SOCIALCRAWL_API_KEY is not configured");

  const url = new URL(endpoint, BASE_URL);
  for (const [name, value] of Object.entries(query)) {
    url.searchParams.set(name, value);
  }

  // Generate once per logical request. Every retry below reuses this value.
  const idempotencyKey = crypto.randomUUID();

  for (let attempt = 0; attempt <= maxRetries; attempt += 1) {
    const startedAt = performance.now();
    const emitAttempt = (details: AttemptDetails) => {
      try {
        onAttempt?.({
          endpoint,
          attempt: attempt + 1,
          latencyMs: Math.round(performance.now() - startedAt),
          ...details,
        });
      } catch {
        // Observability must not change the request outcome.
      }
    };

    const attemptSignal = AbortSignal.timeout(attemptTimeoutMs);
    let response: Response;
    try {
      response = await fetch(url, {
        headers: {
          Accept: "application/json",
          "x-api-key": apiKey,
          "Idempotency-Key": idempotencyKey,
        },
        signal: attemptSignal,
      });
    } catch (cause) {
      const isTimeout =
        attemptSignal.aborted ||
        (cause instanceof DOMException && cause.name === "TimeoutError");
      const details: AttemptDetails = {
        status: null,
        errorType: isTimeout ? "CLIENT_TIMEOUT" : "NETWORK_ERROR",
        requestId: null,
        creditsUsed: null,
        cacheState: null,
      };
      emitAttempt(details);

      if (attempt < maxRetries) {
        await sleep(backoffMs(attempt));
        continue;
      }

      throw new SocialCrawlRequestError(
        isTimeout ? "SocialCrawl request timed out" : "Network request failed",
        {
          endpoint,
          ...details,
          errorType: details.errorType ?? "NETWORK_ERROR",
          attemptCount: attempt + 1,
        },
      );
    }

    const headerRequestId = response.headers.get("x-request-id");
    const retryAfterMs = parseRetryAfter(response.headers.get("retry-after"));
    let body: unknown;
    try {
      body = await response.json();
    } catch {
      const isTimeout = attemptSignal.aborted;
      const details: AttemptDetails = {
        status: response.status,
        errorType: isTimeout ? "CLIENT_TIMEOUT" : "INVALID_RESPONSE",
        requestId: headerRequestId,
        creditsUsed: null,
        cacheState: response.headers.get("x-cache"),
      };
      emitAttempt(details);

      if (
        attempt < maxRetries &&
        (isTimeout || isRetryableHttpFailure(response.status, retryAfterMs))
      ) {
        await sleep(backoffMs(attempt, retryAfterMs));
        continue;
      }

      throw new SocialCrawlRequestError(
        isTimeout
          ? "SocialCrawl response timed out"
          : "Response was not valid JSON",
        {
          endpoint,
          ...details,
          errorType: details.errorType ?? "INVALID_RESPONSE",
          attemptCount: attempt + 1,
        },
      );
    }

    if (!isEnvelope(body)) {
      const partialBody = isRecord(body) ? body : {};
      const partialError = isRecord(partialBody.error) ? partialBody.error : {};
      const details: AttemptDetails = {
        status: response.status,
        errorType:
          typeof partialError.type === "string"
            ? partialError.type
            : "INVALID_RESPONSE",
        requestId:
          typeof partialBody.request_id === "string"
            ? partialBody.request_id
            : headerRequestId,
        creditsUsed:
          typeof partialBody.credits_used === "number"
            ? partialBody.credits_used
            : null,
        cacheState: response.headers.get("x-cache"),
      };
      emitAttempt(details);

      if (
        attempt < maxRetries &&
        isRetryableHttpFailure(response.status, retryAfterMs)
      ) {
        await sleep(backoffMs(attempt, retryAfterMs));
        continue;
      }

      throw new SocialCrawlRequestError("Response envelope was invalid", {
        endpoint,
        ...details,
        errorType: details.errorType ?? "INVALID_RESPONSE",
        attemptCount: attempt + 1,
      });
    }

    const details: AttemptDetails = {
      status: response.status,
      errorType: body.error?.type ?? null,
      requestId: body.request_id,
      creditsUsed: body.credits_used,
      cacheState: response.headers.get("x-cache"),
    };
    emitAttempt(details);

    if (response.ok && body.success) {
      return body as SocialCrawlEnvelope<T>;
    }

    const errorType = body.error?.type ?? "UNKNOWN_ERROR";
    const canRetry = isRetryable(errorType, retryAfterMs);

    if (!canRetry || attempt === maxRetries) {
      throw new SocialCrawlRequestError(body.error?.message ?? errorType, {
        endpoint,
        status: response.status,
        errorType,
        requestId: body.request_id,
        creditsUsed: body.credits_used,
        cacheState: response.headers.get("x-cache"),
        attemptCount: attempt + 1,
      });
    }

    await sleep(backoffMs(attempt, retryAfterMs));
  }

  throw new Error("Unreachable retry state");
}

const profile = await requestSocialCrawl<unknown>({
  endpoint: "/v1/tiktok/profile",
  query: { handle: "charlidamelio" },
  attemptTimeoutMs: 20_000,
  // The callback receives only this allowlisted metadata, never secrets or params.
  onAttempt: (event) => console.info("socialcrawl_attempt", event),
});

console.log({
  requestId: profile.request_id,
  creditsUsed: profile.credits_used,
  cached: profile.cached,
});
```

The example calls the active TikTok profile endpoint with its documented `handle` parameter. It reads the API key only from the server environment. It also keeps `request_id`, charge, and cache state available to your application logs. See [Authentication](/docs/authentication.md) and the full [response schema](/docs/response-schema.md).

## Set timeouts by endpoint

Require an `attemptTimeoutMs` value on every wrapper call. It caps each HTTP attempt, not the total logical request. Choose it from the endpoint's documented behavior and measurements from your own uncached requests. Add enough margin for network latency, then monitor timeout rates after release.

The total retry wall time can be longer than `attemptTimeoutMs`. It is bounded by the maximum attempt count plus the scheduled backoff and any honored `Retry-After` waits. The example's value is illustrative, not a universal timeout. A profile lookup, a paginated feed, and a multi-source search have different latency profiles. Review the selected endpoint page and set a separate per-attempt timeout for each production route.

## Retry only transient failures

Retry only these documented `error.type` values: `IDEMPOTENCY_IN_PROGRESS`, `RATE_LIMITED`, `CONCURRENCY_LIMIT`, `UPSTREAM_ERROR`, `SERVICE_UNAVAILABLE`, and `INTERNAL_ERROR`. Honor `Retry-After` when it is present. Otherwise use bounded exponential backoff with full jitter.

Do not retry deterministic client failures such as invalid authentication, insufficient credits, a key budget breach, invalid parameters, or an idempotency payload mismatch. A withdrawn endpoint returns `503 SERVICE_UNAVAILABLE` permanently without `Retry-After`; surface that response instead of retrying it. See [Error handling](/docs/errors.md) for the complete classification.

## Make retries safe

Generate one unique `Idempotency-Key` for each logical request. Reuse it only when replaying the identical method, path, query parameters, and body. Keys are scoped to the account, and a settled response remains replayable for 24 hours within the documented response-size limit.

An idempotent replay returns the stored response with `credits_used: 0` and `idempotent_replay: true`. Do not reuse the same key for a new page, a changed parameter, or a different operation. Do not send an idempotency key on a GET request that asks for `Accept: text/event-stream`; use JSON for a retriable GET. Review [idempotent requests](/docs/response-schema.md#idempotent-requests) before adding streaming or batch calls.

## Drain every page

For a list response, read the root `pagination.next_cursor` value and send it back unchanged as the next request's `cursor` parameter. Stop when root `pagination.has_more` is `false`.

Pages are sequential because the next cursor comes from the preceding response. Give each page a new idempotency key, but keep that page's key unchanged across its retries. Never infer completion from item count, `total`, or an empty page. See [Pagination](/docs/pagination.md) for the full contract.

## Control request rate and concurrency

Each API key currently has two separate controls: 600 requests per minute and 50 simultaneous requests. A rate limiter controls starts within a time window. A concurrency limiter controls requests still in flight. Track their separate response headers and branch on `RATE_LIMITED` versus `CONCURRENCY_LIMIT` so you reduce the correct pressure.

Set both a worker-pool bound and a request-rate bound below the published ceilings. Apply backpressure before adding retries, since retries consume request capacity too. See [Rate limits](/docs/rate-limits.md) for the headers and current behavior.

## Decide when cached data is acceptable

Use caching only where the endpoint documentation declares it. Read both body `cached` and header `X-Cache` when freshness affects a decision. A reported cache hit uses 0 credits, but a hit is an optimization, not an availability guarantee. An entry can be absent or expired.

Send `Cache-Control: no-cache` only when your application requires a source refresh. On a cache-enabled endpoint, it bypasses the lookup and makes a successful call a billable miss at the endpoint's normal cost. Check [Caching](/docs/caching.md) and the selected endpoint page instead of copying a cache window into application code.

## Separate keys by environment

Create different keys for production, staging, CI, and local development. Keep every key in a server-side secret store and rotate one environment without affecting another.

API keys are not endpoint-scoped permissions. Separate keys provide revocation, usage, rate-limit, and budget isolation, but they do not restrict a key to an endpoint set. Review [Authentication](/docs/authentication.md) before release.

## Cap non-production spend

Set a credit limit on every non-production key. Key limits are cumulative until you reset them, and a key with no configured limit is unscoped by spend. A capped key that reaches its limit returns `KEY_BUDGET_EXCEEDED` without deducting the request cost or stopping other keys on the account.

Inventory every production endpoint and its expected call volume against [Endpoint pricing](/docs/endpoint-pricing.md). Poll `GET /v1/credits/balance` when you need an explicit balance check; the endpoint requires authentication and costs 0 credits. Use [Credits](/docs/credits.md) for ledger semantics rather than assuming a refund or free response before inspecting `credits_used`.

## Configure billing alerts and webhooks

Configure the low-credit email threshold in **Billing > Payments**. In the same screen, configure [billing webhooks](/docs/billing-webhooks.md) for credit and payment events that your infrastructure needs to handle.

Webhook URLs must use HTTPS on a public host. Verify every delivery signature against the raw request body before processing it, and make the handler safe for duplicate delivery. If you also use scheduled Monitor delivery, apply the corresponding [webhook](/docs/webhooks.md) contract.

## Log the fields support needs

Record these fields for every attempt:

- `request_id` from the response body or `X-Request-Id` header
- Endpoint and HTTP status
- `error.type` on failures
- `credits_used`
- Cache state from `cached` or `X-Cache`
- Your measured latency
- Your attempt count

Do not log the API key, idempotency key, webhook secret, or full sensitive request payload. Check [Status](/status) during an incident, and include the affected `request_id` values in a support report so the requests can be traced.

## Deployment checklist

- [ ] Freeze the exact production endpoint set and required parameters.
- [ ] Assign and test a timeout budget for each endpoint.
- [ ] Retry only the documented transient `error.type` values with bounded attempts, `Retry-After`, exponential backoff, and full jitter.
- [ ] Reuse an idempotency key only for an identical retry, and generate a new key for every logical request.
- [ ] Stop pagination on root `pagination.has_more: false`, passing root `next_cursor` back unchanged as `cursor`.
- [ ] Bound request rate and in-flight concurrency separately.
- [ ] Record whether each endpoint permits cached data or requires an explicit refresh.
- [ ] Load a production-only API key from a server-side secret store.
- [ ] Cap every non-production key.
- [ ] Set a credit budget using current endpoint prices and call volume.
- [ ] Configure low-credit alerts and verify webhook signatures against raw bodies.
- [ ] Log `request_id` with status, error type, credits, cache state, latency, and attempt count.
- [ ] Complete a clean pre-production run using [Testing your integration](/docs/testing-your-integration.md).

## Next step

Run the pre-production checks with a separate capped key before you send production traffic.

- [Testing your integration](/docs/testing-your-integration.md): Test success, failure, pagination, retry, cache, and webhook paths before release.
- [Error handling](/docs/errors.md): Use the complete error classification and retry contract.
- [Rate limits](/docs/rate-limits.md): Control request rate and in-flight concurrency separately.
- [Credits](/docs/credits.md): Budget calls and interpret committed credit usage.