Cohorts

Most social listening answers "who is talking about this brand?" Cohorts answers a narrower and usually more valuable question: "which of these specific people are talking about it?"

You bring the list. Upload up to 10,000 platform-qualified public identities, submit a keyword query bounded to a recent window, and get back the matching posts per member — plus an honest coverage record for every member, including the ones that matched nothing.

This is the endpoint to use when you have already decided who matters (a purchaser panel, a customer list, a creator roster, a set of tracked competitors' employees) and you want their public social activity, not the open firehose.

What SocialCrawl receives

Only two things per member:

• platform + handle — the public identity (handle, profile URL, or channel ID)
• external_id — your own opaque key, echoed back on every match so you can join results to your own records

Whatever you used to build the list — purchase receipts, CRM segments, panel attributes, demographics — stays on your side. SocialCrawl never receives it and has no field to put it in.

Both the identity and your external_id are encrypted at rest with a key derived per account. GET /v1/cohorts/{cohortId} returns counts and metadata; it will not read the identities back to you, and they never appear in logs, job payloads, or error messages.

Supported platforms

bluesky · instagram · kwai · linkedin · threads · tiktok · truth-social · twitch · twitter · youtube

Anything else is rejected at upload with 400 COHORT_IDENTITY_PLATFORM_UNSUPPORTED, so an unsupported identity can never silently cost you a query that returns nothing.

The lifecycle

Four calls. Everything except the query itself costs 0 credits.

Every mutating call requires an Idempotency-Key UUID header. Replaying the same key with the same body returns the original resource; replaying it with a different body returns 409.

1. Create the cohort

curl -X POST "https://www.socialcrawl.dev/v1/cohorts" \
  -H "x-api-key: sc_your_api_key_here" \
  -H "Idempotency-Key: 11111111-1111-4111-8111-111111111111" \
  -H "content-type: application/json" \
  -d '{"name":"August purchaser panel","retention_days":30}'

retention_days is 7–90 and defaults to 30. When it elapses, the cohort and everything under it is purged automatically. Uploading members or submitting a query renews the clock.

2. Upload members

Up to 1,000 per call, 10,000 per cohort. Send as many chunks as you need.

curl -X PUT "https://www.socialcrawl.dev/v1/cohorts/{cohortId}/members" \
  -H "x-api-key: sc_your_api_key_here" \
  -H "Idempotency-Key: 22222222-2222-4222-8222-222222222222" \
  -H "content-type: application/json" \
  -d '{
    "members": [
      { "external_id": "buyer_01983", "platform": "instagram", "handle": "natgeo" },
      { "external_id": "buyer_04711", "platform": "youtube",   "handle": "UCxxxxxxxxxxxxxxxxxxxxxx" },
      { "external_id": "buyer_04712", "platform": "linkedin",  "handle": "https://www.linkedin.com/in/someone" }
    ]
  }'

The response reports inserted, updated, unchanged, and the new member_count. Re-sending an external_id you have already uploaded updates its identity rather than adding a row, so a nightly full re-push is safe and does not inflate the count.

Two different external_ids cannot claim the same normalized identity inside one cohort — that returns 409 COHORT_IDENTITY_CONFLICT rather than quietly double-counting one person.

3. Submit a query

curl -X POST "https://www.socialcrawl.dev/v1/cohorts/{cohortId}/queries" \
  -H "x-api-key: sc_your_api_key_here" \
  -H "Idempotency-Key: 33333333-3333-4333-8333-333333333333" \
  -H "content-type: application/json" \
  -d '{
    "keywords": ["acme", "acme pro"],
    "date_from": "2026-08-01T00:00:00.000Z",
    "max_pages_per_identity": 3,
    "max_items_per_identity": 100,
    "max_credits": 12000
  }'

date_from is a full RFC3339 timestamp, not a bare calendar date. platforms is optional — omit it to query every platform present in the cohort, or pass a subset to run one platform at a time.

The call returns 202 immediately:

{
  "query_id": "cqy_3nT7hV2Wq9Ls5Mb8Kd1Rx",
  "status": "queued",
  "member_count": 10000,
  "shard_count": 200,
  "reserved_credits": 8000,
  "estimated_credits": 8000,
  "status_url": "/v1/cohort-queries/cqy_3nT7hV2Wq9Ls5Mb8Kd1Rx",
  "result_url": "/v1/cohort-queries/cqy_3nT7hV2Wq9Ls5Mb8Kd1Rx/results"
}

4. Poll, then read results

curl "https://www.socialcrawl.dev/v1/cohort-queries/{queryId}" \
  -H "x-api-key: sc_your_api_key_here"

status moves through queued → running → succeeded (or failed, cancelled, expired). progress carries durable counters — members_completed, shards_completed, pages_succeeded, pages_failed — so a long run is observable rather than opaque.

Once terminal, read the matches:

curl "https://www.socialcrawl.dev/v1/cohort-queries/{queryId}/results?limit=100" \
  -H "x-api-key: sc_your_api_key_here"

{
  "items": [
    {
      "external_id": "buyer_01983",
      "platform": "instagram",
      "content_id": "3401...",
      "canonical_url": "https://www.instagram.com/p/...",
      "published_at": "2026-08-14T09:12:44.000Z",
      "text_excerpt": "finally switched to acme pro and ...",
      "matched_keywords": ["acme", "acme pro"],
      "route": "profile/posts",
      "retrieved_at": "2026-08-20T11:02:03.000Z"
    }
  ],
  "coverage": [
    {
      "external_id": "buyer_04711",
      "platform": "youtube",
      "status": "complete",
      "pages": 2,
      "oldest_seen": "2026-07-29T00:00:00.000Z",
      "window_complete": true,
      "route_errors": []
    }
  ],
  "next_cursor": "eyJ..."
}

Page with next_cursor until it is null. Pages are additionally bounded so the serialized body never exceeds 1 MB — a page may come back smaller than your limit for that reason, which is not the end of the result set.

Coverage is the important field

coverage has one record per member, whether or not it matched. This is what stops a partial crawl from reading as "nobody talked about you".

• window_complete: true — the route reached your date_from boundary or the account's end of feed. What you got back is everything in the window.
• window_complete: false — the page budget ran out, the account was unreachable, or the route errored. There may be more posts you did not see.
• oldest_seen — the oldest post observed for that member, even when nothing matched. It tells you how far back the crawl actually got.
• route_errors — sanitized per-route failure codes for that member.

A query can be succeeded with pages_failed > 0. Read coverage before treating a result set as exhaustive; raise max_pages_per_identity and re-run if you need to reach the boundary.

Matching is deterministic

Keyword matching is code, not a model. Both the keyword and the candidate text are normalized with Unicode NFKC, case-folded, and whitespace-collapsed, then matched as a literal substring. matched_keywords echoes the keywords you submitted.

There is no stemming, fuzzy matching, semantic expansion, sentiment scoring, or brand-alias inference. If you want "Acme" to also catch "AcmeCo", pass both.

Only content the source attributes to the identity you supplied is eligible. Profile lookups are used to resolve the account and its feed cursor; they never become results.

Credits

Every lifecycle call — create, upload, status, cancel, delete, results — costs 0 credits.

A query reserves its worst-case ceiling at submission:

ceiling = Σ(member × route page cap × credits per page)

Most platforms cost 1 credit per successful upstream page; LinkedIn costs 5. Instagram and YouTube each run two routes per member, so they cost 2 per page-round.

You are then charged only for pages that actually succeeded. Failed, timed-out, cancelled, and skipped pages cost nothing. When the query reaches a terminal state the unspent reservation is refunded exactly once, so actual_credits + refunded_credits always equals reserved_credits.

max_credits is your own safety limit. If the computed ceiling exceeds it, submission fails with 400 before any work is created or any credit is held. It is never permission to spend beyond the ceiling.

Cancelling and deleting

DELETE /v1/cohort-queries/{queryId} stops a query. Queued work never starts; running work stops at the next page boundary. Pages already fetched stay billable, and the unspent reservation is refunded once.

DELETE /v1/cohorts/{cohortId} removes the cohort and cascades its members, queries, and results, cancelling anything still in flight. Your credit-ledger receipts are never deleted — billing history survives the data.

Errors

Any cohort or query that is not yours returns 404 — never 403. A cross-tenant probe is indistinguishable from a resource that does not exist.

Retention and privacy

• Cohort membership is purged after retention_days (7–90, default 30).
• Query results expire no later than their parent cohort, and 30 days after the query goes terminal if that comes sooner.
• Identities and external IDs are encrypted at rest and never appear in logs, job payloads, traces, or error responses.
• Only matched content and coverage are stored — not the unmatched feeds the workers paged through.
• Your GDPR data export includes cohort metadata, query accounting, and your own external_ids. It deliberately excludes the stored platform identities and matched post content.

You remain the controller for how the panel was selected and for the lawful basis of that selection. SocialCrawl processes the public identifiers and public content you submit, for the job you submitted.