# Data Availability (/docs/data-availability)

How to interpret present, null, absent, dropped, cached, and partial data in SocialCrawl responses

A successful response shows what SocialCrawl returned for that request. It does not prove that the source contains no other records.

This illustrative excerpt is not a captured response. It shows the states your parser must handle:

```json title="Response"
{
  "success": true,
  "platform": "tiktok",
  "endpoint": "/v1/tiktok/profile/videos",
  "data": {
    "items": [
      {
        "post": {
          "id": "7527113447973487646",
          "content": { "text": "Mission update" },
          "engagement": { "shares": null }
        }
      }
    ],
    "dropped": 1,
    "_warnings": ["Illustrative advisory message"]
  },
  "cached": false,
  "pagination": {
    "next_cursor": "sc.eyJ2IjoyLCJjIjoiNzM4In0",
    "has_more": true,
    "page_size": 1
  }
}
```

Here, `post.id` and `post.content.text` contain values. `post.engagement.shares` is present with the value `null`. An optional field such as `post.ext.music_id` may be omitted from the object.

## The public-data boundary

SocialCrawl collects social data that is publicly accessible without logging in to the source platform. It does not provide private or login-walled data.

Public availability can change. A source may edit or remove a record, restrict an account, stop exposing a field, or return an incomplete result. A successful request only describes the data available through the selected endpoint at the time and cache state of that response.

## Value, null, absent, dropped, and warning

Treat these as five different states:

| State            | Meaning                                                                                                                 | What your integration should do                                            |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- |
| Value            | The field is present and its value passed the endpoint schema.                                                          | Use the value according to its documented type.                            |
| `null`           | The canonical field exists, but a defensible value is unavailable.                                                      | Preserve `null`. Do not replace it with zero, an empty string, or a guess. |
| Absent           | The field is optional for that schema or response shape and was not included.                                           | Treat the field as optional. Do not assume absence means `null` or zero.   |
| `data.dropped`   | On a list response, source rows were rejected during normalization. The number is the rejected-row count for that page. | Read it on every page and record or alert on unexpected non-zero values.   |
| `data._warnings` | The response has an advisory partial-data or transformation condition.                                                  | Keep the successful response, then log and inspect each warning.           |

A successful response with `data._warnings` remains valid. Warnings do not turn it into an error response. See [Response schema](/docs/response-schema.md) for the envelope contract and [Computed fields](/docs/computed-fields.md) for the rules that produce computed values or `null`.

## A page is not the complete set

One list page proves only what that page contains. Inspect `data.dropped`, then follow the top-level pagination block when it is present.

Send `pagination.next_cursor` back unchanged as the `cursor` query parameter. Stop when `pagination.has_more` is `false`. Never use `data.total` as the stop condition. The [Pagination guide](/docs/pagination.md) defines the full loop.

Reaching the final page means you fetched every page the endpoint made available. It does not prove complete historical coverage. Endpoint depth, source availability, deleted records, and access restrictions can still limit the set.

## Do not depend on source ordering

The source can change the order of records between requests. Do not treat a row's position as a stable identifier or assume that the first page always covers the same time range.

Use documented sort parameters when the endpoint provides them. Store canonical IDs and timestamps from the response, and deduplicate across pages and runs where your workflow requires it. Check the selected endpoint's documentation before relying on any ordering rule.

## Geography, language, and date limits

Geography, language, date filters, result depth, and coverage vary by endpoint. A parameter supported by one search or feed endpoint is not a platform-wide guarantee.

Read the selected endpoint's declared parameters before you build a query. If the endpoint does not document a region, language, date, or depth filter, do not assume it applies one. Filter returned fields on your side only when those fields are present and suitable for the decision.

For example, review the generated [TikTok API reference](/docs/api-reference.md#tag/tiktok) before using `/v1/tiktok/profile/videos`. It is the current source for that endpoint's accepted parameters and response contract.

## Freshness and cache state

Use the body field `cached` and the `X-Cache` response header to identify the cache state of a response. A cache hit is uncharged, but a hit is never guaranteed. Cache behavior applies only where the endpoint documentation says the endpoint is cache-enabled.

If your decision requires a new source fetch, send `Cache-Control: no-cache`. This forces a billable cache miss at the endpoint's normal cost. `Cache-Control: no-store` does not bypass the cache on its own. Read [Caching](/docs/caching.md) for the complete cache contract and endpoint-specific guidance.

Freshness does not establish completeness. A new source fetch can still reflect data that the source changed, removed, restricted, or did not expose.

## Provenance and extension fields

Keep the response's `platform`, `endpoint`, and `request_id` with the canonical IDs and source URLs you use. These fields help you trace which request produced a stored record.

Fields under `ext.*` carry source-specific details and join keys. Treat every extension field as optional. Use the generated [Post schema](/docs/schema/post.md) or the matching object schema to confirm its type and availability.

Published schema versions remain backwards compatible. New optional fields can appear, so consumers must ignore unknown fields instead of rejecting the whole response. [Schema compatibility](/docs/schema-compatibility.md) explains additive changes and version boundaries.

## Choose the endpoint that matches the question

Endpoint choice sets the practical availability boundary. A profile lookup, post list, keyword search, comment list, and cross-platform search answer different questions and may expose different filters and depth.

Start with the question you need to answer. Then use [Choosing endpoints](/docs/choosing-endpoints.md) to select the narrowest endpoint that returns the required object, coverage, and controls. Confirm its current parameters and response fields in the [API reference](/docs/api-reference.md).

## Endpoint-fit checklist

Before you depend on an endpoint, confirm:

- The endpoint returns the object and fields your question requires.
- Its public-data boundary matches the records you expect to collect.
- Its geography, language, date, ordering, and depth controls are documented for that endpoint.
- Your list loop uses `pagination.next_cursor` unchanged and stops on `pagination.has_more`.
- You inspect `data.dropped` on every list page and log `data._warnings`.
- Your parser distinguishes a value, `null`, and an absent optional field.
- Your freshness policy checks `cached` or `X-Cache` and budgets for billable forced misses.
- Your stored records keep canonical IDs and enough request context to trace their source.
- Your consumer ignores unknown additive fields while validating fields it uses.

Next, use [Choosing endpoints](/docs/choosing-endpoints.md) to match your first production question to the right endpoint.

- [Choosing endpoints](/docs/choosing-endpoints.md): Select the endpoint whose scope and controls match your question.
- [Response schema](/docs/response-schema.md): Handle canonical values, nulls, dropped rows, and warnings.
- [Pagination](/docs/pagination.md): Drain list pages with the canonical cursor and stop signal.
- [Caching](/docs/caching.md): Read cache state and decide when a billable fresh fetch is required.