Cohorts

Cohorts answers a narrower question than open social listening: which of these specific people are talking about it? You upload up to 10,000 platform-qualified public identities, submit a keyword query bounded to a recent window, and get back the matching posts per member plus a coverage record for every member, including the ones that matched nothing.

Reach for it when you have already decided who matters (a purchaser panel, a customer list, a creator roster) and you want their public social activity rather than the open firehose. Queries run asynchronously, stay private to your account, and bill per successful upstream page.

What SocialCrawl receives

Only two things per member:

• platform + handle, the public identity (the account handle, or the profile URL for LinkedIn).
• external_id, your own opaque key. It is echoed back on every match so you can join results to your own records.

Whatever you used to build the list (purchase receipts, CRM segments, panel attributes, demographics) stays on your side. SocialCrawl never receives it and has no field to put it in.

Both the identity and your external_id are encrypted at rest with a key derived per account. GET /v1/cohorts/{cohortId} returns counts and metadata. It will not read the identities back to you, and they never appear in logs, job payloads, or error messages.

Supported platforms

bluesky · instagram · kwai · linkedin · threads · tiktok · truth-social · twitch · twitter · youtube

Anything else is rejected at upload with 400 COHORT_IDENTITY_PLATFORM_UNSUPPORTED, so an unsupported identity can never silently cost you a query that returns nothing.

The lifecycle

Four calls. Everything except the query itself costs 0 credits.

POST and PUT calls require an Idempotency-Key UUID header. Replaying the same key with the same body returns the original resource. Replaying it with a different body returns 422 IDEMPOTENCY_KEY_PAYLOAD_MISMATCH. DELETE needs no key.

1. Create the cohort

curl -X POST "https://www.socialcrawl.dev/v1/cohorts" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Idempotency-Key: 11111111-1111-4111-8111-111111111111" \
  -H "content-type: application/json" \
  -d '{"name":"August purchaser panel","retention_days":30}'

retention_days is 7 to 90 and defaults to 30. When it elapses, the cohort and everything under it is purged automatically. Uploading members or submitting a query renews the clock.

2. Upload members

Up to 1,000 per call, 10,000 per cohort. Send as many chunks as you need.

curl -X PUT "https://www.socialcrawl.dev/v1/cohorts/{cohortId}/members" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Idempotency-Key: 22222222-2222-4222-8222-222222222222" \
  -H "content-type: application/json" \
  -d '{
    "members": [
      { "external_id": "buyer_01983", "platform": "instagram", "handle": "natgeo" },
      { "external_id": "buyer_04711", "platform": "youtube",   "handle": "mkbhd" },
      { "external_id": "buyer_04712", "platform": "linkedin",  "handle": "https://www.linkedin.com/in/someone" }
    ]
  }'

The response reports inserted, updated, unchanged, and the new member_count. Re-sending an external_id you have already uploaded updates its identity rather than adding a row, so a nightly full re-push is safe and does not inflate the count.

Two different external_ids cannot claim the same normalized identity inside one cohort. That returns 409 COHORT_IDENTITY_CONFLICT rather than quietly double-counting one person.

3. Submit a query

curl -X POST "https://www.socialcrawl.dev/v1/cohorts/{cohortId}/queries" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Idempotency-Key: 33333333-3333-4333-8333-333333333333" \
  -H "content-type: application/json" \
  -d '{
    "keywords": ["acme", "acme pro"],
    "date_from": "2026-08-01T00:00:00.000Z",
    "max_pages_per_identity": 3,
    "max_items_per_identity": 100,
    "max_credits": 30000
  }'

• keywordsstring[]required

  Terms to match. Matching is literal and whole-word, with no stemming or expansion.

• date_fromstringrequired

  A full RFC3339 timestamp, not a bare calendar date. It bounds how far back each crawl reaches.

• max_pages_per_identityintegerrequired

  Page budget per member. No default. Single-page platforms ignore anything above 1.

• max_items_per_identityintegerrequired

  Item budget per member. No default.

• max_creditsintegerrequired

  Your own safety limit. Submission fails with a 400 if the computed ceiling exceeds it, before any credit is held.

• platformsstring[]

  Omit to query every platform present in the cohort, or pass a subset to run one platform at a time.

All three caps are required and none of them has a default. max_credits must cover the worst-case ceiling or submission fails before any credit is held. The example above is a 10,000-member panel entirely on cursor-paginated 1-credit-per-page platforms at 3 pages each, so its ceiling is 30,000.

The call returns 202 immediately:

{
  "query_id": "ziJygy91eDlzFJLzzUgIY",
  "status": "queued",
  "member_count": 10000,
  "shard_count": 200,
  "reserved_credits": 30000,
  "estimated_credits": 30000,
  "status_url": "/v1/cohort-queries/ziJygy91eDlzFJLzzUgIY",
  "result_url": "/v1/cohort-queries/ziJygy91eDlzFJLzzUgIY/results"
}

4. Poll, then read results

curl "https://www.socialcrawl.dev/v1/cohort-queries/{queryId}" \
  -H "x-api-key: YOUR_API_KEY"

status moves through queued → running → succeeded (or failed, cancelled, expired). progress carries durable counters (members_completed, shards_completed, pages_succeeded, pages_failed), so a long run is observable rather than opaque.

Once the query has succeeded, read the matches (a cancelled or expired query does not serve results):

curl "https://www.socialcrawl.dev/v1/cohort-queries/{queryId}/results?limit=100" \
  -H "x-api-key: YOUR_API_KEY"

{
  "items": [
    {
      "external_id": "buyer_01983",
      "platform": "instagram",
      "content_id": "3401...",
      "canonical_url": "https://www.instagram.com/p/...",
      "published_at": "2026-08-14T09:12:44.000Z",
      "text_excerpt": "finally switched to acme pro and ...",
      "matched_keywords": ["acme", "acme pro"],
      "route": "profile/posts",
      "retrieved_at": "2026-08-20T11:02:03.000Z"
    }
  ],
  "coverage": [
    {
      "external_id": "buyer_04711",
      "platform": "youtube",
      "status": "complete",
      "pages": 2,
      "oldest_seen": "2026-07-29T00:00:00.000Z",
      "window_complete": true,
      "route_errors": []
    }
  ],
  "next_cursor": "eyJ..."
}

Page with next_cursor until it is null. coverage is paginated exactly like items: each page carries up to limit coverage records, so a 10,000-member panel takes at least 20 pages at limit=500 even when nothing matched. Later pages can arrive with an empty items array that still carries coverage, so accumulate both streams until the cursor is null.

Pages are additionally bounded so the serialized body never exceeds 1 MB. A page that comes back smaller than your limit for that reason is not the end of the result set.

Coverage is the important field

coverage has one record per member, whether or not it matched. This is what stops a partial crawl from reading as "nobody talked about you".

• window_complete: true, the route reached your date_from boundary or the account's end of feed. What you got back is everything in the window.
• window_complete: false, the page budget ran out, the account was unreachable, or the route errored. There may be more posts you did not see.
• oldest_seen, the oldest post observed for that member, even when nothing matched. It tells you how far back the crawl actually got.
• status per member is the failure signal to key off: a handle that does not resolve reports not_found, and an exhausted or erroring account reports failed.

A query can be succeeded while individual members report not_found or failed. Read coverage before treating a result set as exhaustive; raise max_pages_per_identity and re-run if you need to reach the boundary. On the single-page platforms (X/Twitter, Bluesky, Threads, Twitch) the source serves one fixed page per identity, so raising the cap cannot deepen the crawl there. window_complete: false on those platforms means the window is simply beyond what the source exposes.

Matching is deterministic

Keyword matching is code, not a model. The exact rules:

• Normalization. Both the keyword and the candidate text are normalized with Unicode NFKC, case-folded, and whitespace-collapsed.
• Whole-word only. The keyword must not be immediately joined to another letter, digit, or mark on either side. acme matches "my acme review" but not "acmecorp".
• CJK caveat. CJK scripts count as letters, so a Korean, Japanese, or Chinese keyword embedded directly inside surrounding CJK text will not match. Include the surrounding form as its own keyword when you need it.
• Echoed normalized. matched_keywords reports your keywords in their normalized form. Submit "Acme Pro" and a match reports "acme pro".
• Title and description. On YouTube and Twitch the matched text covers both the title and the description, and text_excerpt carries whichever surface matched.

There is no stemming, fuzzy matching, semantic expansion, sentiment scoring, or brand-alias inference. If you want "Acme" to also catch "AcmeCo", pass both.

Only content the source attributes to the identity you supplied is eligible. Profile lookups resolve the account and its feed cursor. They never become results.

Credits

Every lifecycle call (create, upload, status, cancel, delete, results) costs 0 credits.

A query reserves its worst-case ceiling at submission:

ceiling = Σ(member × route page cap × credits per page)

The single-page platforms cannot be paged past their first response, so their page cap in the ceiling is 1 regardless of max_pages_per_identity. The reservation never exceeds what the query can actually spend.

You are then charged only for pages that actually succeeded. Failed, timed-out, cancelled, and skipped pages cost nothing. When the query reaches a terminal state, the unspent reservation is refunded exactly once, so actual_credits + refunded_credits always equals reserved_credits.

max_credits is your own safety limit. If the computed ceiling exceeds it, submission fails with a 400 before any work is created or any credit is held. It is never permission to spend beyond the ceiling.

Cancelling and deleting

DELETE /v1/cohort-queries/{queryId} stops a query. Queued work never starts; running work stops at the next page boundary. Pages already fetched stay billable, and the unspent reservation is refunded once.

DELETE /v1/cohorts/{cohortId} removes the cohort and cascades its members, queries, and results, cancelling anything still in flight. Your credit-ledger receipts are never deleted. Billing history survives the data.

Errors

Any cohort or query that is not yours returns 404, never 403. A cross-tenant probe is indistinguishable from a resource that does not exist. Full detail on each code is on the Error handling page.

Retention and privacy

• Cohort membership is purged after retention_days (7 to 90, default 30).
• Query results are purged together with their parent cohort when its retention elapses.
• Identities and external IDs are encrypted at rest and never appear in logs, job payloads, traces, or error responses.
• Only matched content and coverage are stored, not the unmatched feeds the workers paged through.
• Your GDPR data export includes cohort metadata, query accounting, and your own external_ids. It deliberately excludes the stored platform identities and matched post content.

You remain the controller for how the panel was selected and for the lawful basis of that selection. SocialCrawl processes the public identifiers and public content you submit, for the job you submitted.

Cookbook: a 10,000-identity panel end to end

At full panel size you want the four calls wrapped in a loader you can re-run without fear. The script below is complete and safe to run twice. Every mutating call derives its Idempotency-Key from the request itself (UUIDv5 over the operation and body), so a crashed or repeated run replays the calls it already made instead of creating a second cohort or reserving a second query. Member upload is an upsert on external_id, so re-pushing the panel is safe on its own.

It reads a CSV with external_id,platform,handle columns, loads the panel in chunks of 1,000, submits one bounded query, polls with backoff, then writes matches.ndjson and coverage.csv. Every match and every coverage row carries your external_id, so joining back to your own records is a plain merge.

Operational notes for a recurring pipeline

• The cohort is reusable. Upload the panel once, then run as many queries against it as you like: one per brand, one per keyword set, one per reporting window. Each query bills only its own pages, and querying renews the retention clock.
• Refresh, do not rebuild. A nightly re-push of the full panel is an upsert keyed on external_id. Changed handles update, unchanged rows do nothing, and the member count never inflates.
• Keep chunk order stable across retries. The idempotency fingerprint preserves array order, so replaying the same members in a different order under the same key is a 409. Build chunks from an ordered source such as a file or a sorted query, never from a set or an unordered map.
• Segment with cohorts, not filters. One cohort per purchaser segment keeps each query's cost proportional to the segment, and keeps your segmentation logic on your side.
• Read coverage before concluding silence. A member with status: "complete" and no matches genuinely did not post about you. A member with window_complete: false was not fully crawled. Those are different findings.

Next steps