Data Availability

A successful response shows what SocialCrawl returned for that request. It does not prove that the source contains no other records.

This illustrative excerpt is not a captured response. It shows the states your parser must handle:

{
  "success": true,
  "platform": "tiktok",
  "endpoint": "/v1/tiktok/profile/videos",
  "data": {
    "items": [
      {
        "post": {
          "id": "7527113447973487646",
          "content": { "text": "Mission update" },
          "engagement": { "shares": null }
        }
      }
    ],
    "dropped": 1,
    "_warnings": ["Illustrative advisory message"]
  },
  "cached": false,
  "pagination": {
    "next_cursor": "sc.eyJ2IjoyLCJjIjoiNzM4In0",
    "has_more": true,
    "page_size": 1
  }
}

Here, post.id and post.content.text contain values. post.engagement.shares is present with the value null. An optional field such as post.ext.music_id may be omitted from the object.

The public-data boundary

SocialCrawl collects social data that is publicly accessible without logging in to the source platform. It does not provide private or login-walled data.

Public availability can change. A source may edit or remove a record, restrict an account, stop exposing a field, or return an incomplete result. A successful request only describes the data available through the selected endpoint at the time and cache state of that response.

Value, null, absent, dropped, and warning

Treat these as five different states:

A successful response with data._warnings remains valid. Warnings do not turn it into an error response. See Response schema for the envelope contract and Computed fields for the rules that produce computed values or null.

A page is not the complete set

One list page proves only what that page contains. Inspect data.dropped, then follow the top-level pagination block when it is present.

Send pagination.next_cursor back unchanged as the cursor query parameter. Stop when pagination.has_more is false. Never use data.total as the stop condition. The Pagination guide defines the full loop.

Reaching the final page means you fetched every page the endpoint made available. It does not prove complete historical coverage. Endpoint depth, source availability, deleted records, and access restrictions can still limit the set.

Do not depend on source ordering

The source can change the order of records between requests. Do not treat a row's position as a stable identifier or assume that the first page always covers the same time range.

Use documented sort parameters when the endpoint provides them. Store canonical IDs and timestamps from the response, and deduplicate across pages and runs where your workflow requires it. Check the selected endpoint's documentation before relying on any ordering rule.

Geography, language, and date limits

Geography, language, date filters, result depth, and coverage vary by endpoint. A parameter supported by one search or feed endpoint is not a platform-wide guarantee.

Read the selected endpoint's declared parameters before you build a query. If the endpoint does not document a region, language, date, or depth filter, do not assume it applies one. Filter returned fields on your side only when those fields are present and suitable for the decision.

For example, review the generated TikTok API reference before using /v1/tiktok/profile/videos. It is the current source for that endpoint's accepted parameters and response contract.

Freshness and cache state

Use the body field cached and the X-Cache response header to identify the cache state of a response. A cache hit is uncharged, but a hit is never guaranteed. Cache behavior applies only where the endpoint documentation says the endpoint is cache-enabled.

If your decision requires a new source fetch, send Cache-Control: no-cache. This forces a billable cache miss at the endpoint's normal cost. Cache-Control: no-store does not bypass the cache on its own. Read Caching for the complete cache contract and endpoint-specific guidance.

Freshness does not establish completeness. A new source fetch can still reflect data that the source changed, removed, restricted, or did not expose.

Provenance and extension fields

Keep the response's platform, endpoint, and request_id with the canonical IDs and source URLs you use. These fields help you trace which request produced a stored record.

Fields under ext.* carry source-specific details and join keys. Treat every extension field as optional. Use the generated Post schema or the matching object schema to confirm its type and availability.

Published schema versions remain backwards compatible. New optional fields can appear, so consumers must ignore unknown fields instead of rejecting the whole response. Schema compatibility explains additive changes and version boundaries.

Choose the endpoint that matches the question

Endpoint choice sets the practical availability boundary. A profile lookup, post list, keyword search, comment list, and cross-platform search answer different questions and may expose different filters and depth.

Start with the question you need to answer. Then use Choosing endpoints to select the narrowest endpoint that returns the required object, coverage, and controls. Confirm its current parameters and response fields in the API reference.

Endpoint-fit checklist

Before you depend on an endpoint, confirm:

• The endpoint returns the object and fields your question requires.
• Its public-data boundary matches the records you expect to collect.
• Its geography, language, date, ordering, and depth controls are documented for that endpoint.
• Your list loop uses pagination.next_cursor unchanged and stops on pagination.has_more.
• You inspect data.dropped on every list page and log data._warnings.
• Your parser distinguishes a value, null, and an absent optional field.
• Your freshness policy checks cached or X-Cache and budgets for billable forced misses.
• Your stored records keep canonical IDs and enough request context to trace their source.
• Your consumer ignores unknown additive fields while validating fields it uses.

Next, use Choosing endpoints to match your first production question to the right endpoint.