# Cohorts (/docs/cohorts) Upload up to 10,000 public social identities, then ask which of them posted about your keywords in a recent window Cohorts answers a narrower question than open social listening: **which of _these specific people_ are talking about it?** You upload up to 10,000 platform-qualified public identities, submit a keyword query bounded to a recent window, and get back the matching posts per member plus a coverage record for every member, including the ones that matched nothing. Reach for it when you have already decided who matters (a purchaser panel, a customer list, a creator roster) and you want their public social activity rather than the open firehose. Queries run asynchronously, stay private to your account, and bill per successful upstream page. Cohorts is **not** audience discovery. It does not find people, infer demographics, or score interests. It takes a list you already have and tells you what those public accounts posted. ## What SocialCrawl receives Only two things per member: - `platform` + `handle`, the public identity (the account handle, or the profile URL for LinkedIn). - `external_id`, your own opaque key. It is echoed back on every match so you can join results to your own records. Whatever you used to build the list (purchase receipts, CRM segments, panel attributes, demographics) stays on your side. SocialCrawl never receives it and has no field to put it in. Both the identity and your `external_id` are encrypted at rest with a key derived per account. `GET /v1/cohorts/{cohortId}` returns counts and metadata. It will not read the identities back to you, and they never appear in logs, job payloads, or error messages. ## Supported platforms `bluesky` · `instagram` · `kwai` · `linkedin` · `threads` · `tiktok` · `truth-social` · `twitch` · `twitter` · `youtube` Anything else is rejected at upload with 400 `COHORT_IDENTITY_PLATFORM_UNSUPPORTED`, so an unsupported identity can never silently cost you a query that returns nothing. ## The lifecycle Four calls. Everything except the query itself costs 0 credits. | Step | Call | Cost | | ---- | ------------------------------------------------------- | ------- | | 1 | `POST /v1/cohorts` | 0 | | 2 | `PUT /v1/cohorts/{cohortId}/members` (repeat per 1,000) | 0 | | 3 | `POST /v1/cohorts/{cohortId}/queries` → `202` | metered | | 4 | `GET /v1/cohort-queries/{queryId}` then `.../results` | 0 | `POST` and `PUT` calls require an `Idempotency-Key` UUID header. Replaying the same key with the same body returns the original resource. Replaying it with a different body returns 422 `IDEMPOTENCY_KEY_PAYLOAD_MISMATCH`. `DELETE` needs no key. ### 1. Create the cohort ```bash title="cURL" curl -X POST "https://www.socialcrawl.dev/v1/cohorts" \ -H "x-api-key: YOUR_API_KEY" \ -H "Idempotency-Key: 11111111-1111-4111-8111-111111111111" \ -H "content-type: application/json" \ -d '{"name":"August purchaser panel","retention_days":30}' ``` `retention_days` is 7 to 90 and defaults to 30. When it elapses, the cohort and everything under it is purged automatically. Uploading members or submitting a query renews the clock. ### 2. Upload members Up to 1,000 per call, 10,000 per cohort. Send as many chunks as you need. ```bash title="cURL" curl -X PUT "https://www.socialcrawl.dev/v1/cohorts/{cohortId}/members" \ -H "x-api-key: YOUR_API_KEY" \ -H "Idempotency-Key: 22222222-2222-4222-8222-222222222222" \ -H "content-type: application/json" \ -d '{ "members": [ { "external_id": "buyer_01983", "platform": "instagram", "handle": "natgeo" }, { "external_id": "buyer_04711", "platform": "youtube", "handle": "mkbhd" }, { "external_id": "buyer_04712", "platform": "linkedin", "handle": "https://www.linkedin.com/in/someone" } ] }' ``` The response reports `inserted`, `updated`, `unchanged`, and the new `member_count`. Re-sending an `external_id` you have already uploaded **updates** its identity rather than adding a row, so a nightly full re-push is safe and does not inflate the count. Two different `external_id`s cannot claim the same normalized identity inside one cohort. That returns 409 `COHORT_IDENTITY_CONFLICT` rather than quietly double-counting one person. ### 3. Submit a query ```bash title="cURL" curl -X POST "https://www.socialcrawl.dev/v1/cohorts/{cohortId}/queries" \ -H "x-api-key: YOUR_API_KEY" \ -H "Idempotency-Key: 33333333-3333-4333-8333-333333333333" \ -H "content-type: application/json" \ -d '{ "keywords": ["acme", "acme pro"], "date_from": "2026-08-01T00:00:00.000Z", "max_pages_per_identity": 3, "max_items_per_identity": 100, "max_credits": 30000 }' ``` | Field | Type | Required | Description | | --- | --- | --- | --- | | `keywords` | `string[]` | yes | Terms to match. Matching is literal and whole-word, with no stemming or expansion. | | `date_from` | `string` | yes | A full RFC3339 timestamp, not a bare calendar date. It bounds how far back each crawl reaches. | | `max_pages_per_identity` | `integer` | yes | Page budget per member. No default. Single-page platforms ignore anything above 1. | | `max_items_per_identity` | `integer` | yes | Item budget per member. No default. | | `max_credits` | `integer` | yes | Your own safety limit. Submission fails with a 400 if the computed ceiling exceeds it, before any credit is held. | | `platforms` | `string[]` | no | Omit to query every platform present in the cohort, or pass a subset to run one platform at a time. | All three caps are required and none of them has a default. `max_credits` must cover the worst-case ceiling or submission fails before any credit is held. The example above is a 10,000-member panel entirely on cursor-paginated 1-credit-per-page platforms at 3 pages each, so its ceiling is 30,000. The call returns `202` immediately: ```json title="JSON" { "query_id": "ziJygy91eDlzFJLzzUgIY", "status": "queued", "member_count": 10000, "shard_count": 200, "reserved_credits": 30000, "estimated_credits": 30000, "status_url": "/v1/cohort-queries/ziJygy91eDlzFJLzzUgIY", "result_url": "/v1/cohort-queries/ziJygy91eDlzFJLzzUgIY/results" } ``` ### 4. Poll, then read results ```bash title="cURL" curl "https://www.socialcrawl.dev/v1/cohort-queries/{queryId}" \ -H "x-api-key: YOUR_API_KEY" ``` `status` moves through `queued` → `running` → `succeeded` (or `failed`, `cancelled`, `expired`). `progress` carries durable counters (`members_completed`, `shards_completed`, `pages_succeeded`, `pages_failed`), so a long run is observable rather than opaque. Once the query has **succeeded**, read the matches (a cancelled or expired query does not serve results): ```bash title="cURL" curl "https://www.socialcrawl.dev/v1/cohort-queries/{queryId}/results?limit=100" \ -H "x-api-key: YOUR_API_KEY" ``` ```json title="JSON" { "items": [ { "external_id": "buyer_01983", "platform": "instagram", "content_id": "3401...", "canonical_url": "https://www.instagram.com/p/...", "published_at": "2026-08-14T09:12:44.000Z", "text_excerpt": "finally switched to acme pro and ...", "matched_keywords": ["acme", "acme pro"], "route": "profile/posts", "retrieved_at": "2026-08-20T11:02:03.000Z" } ], "coverage": [ { "external_id": "buyer_04711", "platform": "youtube", "status": "complete", "pages": 2, "oldest_seen": "2026-07-29T00:00:00.000Z", "window_complete": true, "route_errors": [] } ], "next_cursor": "eyJ..." } ``` Page with `next_cursor` until it is `null`. `coverage` is paginated exactly like `items`: each page carries up to `limit` coverage records, so a 10,000-member panel takes at least 20 pages at `limit=500` even when nothing matched. Later pages can arrive with an empty `items` array that still carries coverage, so accumulate both streams until the cursor is `null`. Pages are additionally bounded so the serialized body never exceeds 1 MB. A page that comes back smaller than your `limit` for that reason is not the end of the result set. ## Coverage is the important field `coverage` has one record per member, whether or not it matched. This is what stops a partial crawl from reading as "nobody talked about you". - `window_complete: true`, the route reached your `date_from` boundary or the account's end of feed. What you got back is everything in the window. - `window_complete: false`, the page budget ran out, the account was unreachable, or the route errored. There may be more posts you did not see. - `oldest_seen`, the oldest post observed for that member, even when nothing matched. It tells you how far back the crawl actually got. - `status` per member is the failure signal to key off: a handle that does not resolve reports `not_found`, and an exhausted or erroring account reports `failed`. A query can be `succeeded` while individual members report `not_found` or `failed`. Read `coverage` before treating a result set as exhaustive; raise `max_pages_per_identity` and re-run if you need to reach the boundary. On the single-page platforms (X/Twitter, Bluesky, Threads, Twitch) the source serves one fixed page per identity, so raising the cap cannot deepen the crawl there. `window_complete: false` on those platforms means the window is simply beyond what the source exposes. ## Matching is deterministic Keyword matching is code, not a model. The exact rules: - **Normalization.** Both the keyword and the candidate text are normalized with Unicode NFKC, case-folded, and whitespace-collapsed. - **Whole-word only.** The keyword must not be immediately joined to another letter, digit, or mark on either side. `acme` matches "my acme review" but not "acmecorp". - **CJK caveat.** CJK scripts count as letters, so a Korean, Japanese, or Chinese keyword embedded directly inside surrounding CJK text will not match. Include the surrounding form as its own keyword when you need it. - **Echoed normalized.** `matched_keywords` reports your keywords in their normalized form. Submit "Acme Pro" and a match reports `"acme pro"`. - **Title and description.** On YouTube and Twitch the matched text covers both the title and the description, and `text_excerpt` carries whichever surface matched. There is no stemming, fuzzy matching, semantic expansion, sentiment scoring, or brand-alias inference. If you want "Acme" to also catch "AcmeCo", pass both. Only content the source attributes to the identity you supplied is eligible. Profile lookups resolve the account and its feed cursor. They never become results. ## Credits Every lifecycle call (create, upload, status, cancel, delete, results) costs **0 credits**. A query reserves its worst-case ceiling at submission: ```text title="Formula" ceiling = Σ(member × route page cap × credits per page) ``` | Platform | Credits per successful page | | ------------------------------------- | ------------------------------------------------------ | | LinkedIn | 5 | | Instagram, YouTube | 2 per page-round (each runs two routes per member) | | X (Twitter), Bluesky, Threads, Twitch | 1, and the page cap is always 1 (they cannot be paged) | | Everything else | 1 | The single-page platforms cannot be paged past their first response, so their page cap in the ceiling is 1 regardless of `max_pages_per_identity`. The reservation never exceeds what the query can actually spend. You are then charged **only for pages that actually succeeded**. Failed, timed-out, cancelled, and skipped pages cost nothing. When the query reaches a terminal state, the unspent reservation is refunded exactly once, so `actual_credits + refunded_credits` always equals `reserved_credits`. `max_credits` is your own safety limit. If the computed ceiling exceeds it, submission fails with a `400` before any work is created or any credit is held. It is never permission to spend beyond the ceiling. ## Cancelling and deleting `DELETE /v1/cohort-queries/{queryId}` stops a query. Queued work never starts; running work stops at the next page boundary. Pages already fetched stay billable, and the unspent reservation is refunded once. `DELETE /v1/cohorts/{cohortId}` removes the cohort and cascades its members, queries, and results, cancelling anything still in flight. Your credit-ledger receipts are never deleted. Billing history survives the data. ## Errors | Code | Status | Meaning | | -------------------------------------- | ------ | ----------------------------------------------------------------------------------------- | | `COHORT_LIMIT_EXCEEDED` | 400 | The account already holds the maximum of 100 cohorts | | `COHORT_MEMBER_LIMIT_EXCEEDED` | 400 | The upload would push the cohort past 10,000 members | | `COHORT_IDENTITY_PLATFORM_UNSUPPORTED` | 400 | That platform is not supported for cohort queries | | `COHORT_IDENTITY_CONFLICT` | 409 | Another `external_id` already claims that identity in this cohort | | `COHORT_QUERY_NOT_READY` | 409 | The query has not succeeded. Running, cancelled, and expired queries do not serve results | | `COHORT_QUERY_NOT_CANCELLABLE` | 409 | The query is already terminal | | `COHORT_RESULT_TOO_LARGE` | 413 | A single stored result cannot fit under the 1 MB page ceiling | | `INSUFFICIENT_CREDITS` | 402 | The reservation ceiling exceeds your available balance | Any cohort or query that is not yours returns `404`, never `403`. A cross-tenant probe is indistinguishable from a resource that does not exist. Full detail on each code is on the [Error handling](/docs/errors.md) page. ## Retention and privacy - Cohort membership is purged after `retention_days` (7 to 90, default 30). - Query results are purged together with their parent cohort when its retention elapses. - Identities and external IDs are encrypted at rest and never appear in logs, job payloads, traces, or error responses. - Only matched content and coverage are stored, not the unmatched feeds the workers paged through. - Your GDPR data export includes cohort metadata, query accounting, and your own `external_id`s. It deliberately excludes the stored platform identities and matched post content. You remain the controller for how the panel was selected and for the lawful basis of that selection. SocialCrawl processes the public identifiers and public content you submit, for the job you submitted. ## Cookbook: a 10,000-identity panel end to end At full panel size you want the four calls wrapped in a loader you can re-run without fear. The script below is complete and safe to run twice. Every mutating call derives its `Idempotency-Key` from the request itself (UUIDv5 over the operation and body), so a crashed or repeated run replays the calls it already made instead of creating a second cohort or reserving a second query. Member upload is an upsert on `external_id`, so re-pushing the panel is safe on its own. It reads a CSV with `external_id,platform,handle` columns, loads the panel in chunks of 1,000, submits one bounded query, polls with backoff, then writes `matches.ndjson` and `coverage.csv`. Every match and every coverage row carries your `external_id`, so joining back to your own records is a plain merge. ```python title="Python" #!/usr/bin/env python3 """Run a SocialCrawl cohort query over a panel CSV, end to end. Usage: SOCIALCRAWL_API_KEY=sc_... python cohort_panel_runner.py \ panel.csv --name "August purchaser panel" \ --keywords "acme" "acme pro" --days 30 """ import argparse, csv, hashlib, json, sys, time, unicodedata, uuid from datetime import datetime, timedelta, timezone import requests BASE = "https://www.socialcrawl.dev" NAMESPACE = uuid.NAMESPACE_DNS CHUNK = 1_000 CREDITS_PER_PAGE = { # Credits per successful page. LinkedIn is 5; Instagram and YouTube run # two routes per member (a page-round costs 2); every other platform is 1. "linkedin": 5, "instagram": 2, "youtube": 2, } def credit_ceiling(panel, max_pages: int) -> int: """Safe upper bound on the reservation. The API reserves at most this; single-page platforms (twitter, bluesky, threads, twitch) actually reserve one page each, so a max_credits at this value always clears submission. """ return sum( CREDITS_PER_PAGE.get(row["platform"], 1) * max_pages for row in panel ) def idem_key(*parts: str) -> str: """Deterministic Idempotency-Key: same request, same key, safe re-runs.""" material = "\x1f".join(parts) return str(uuid.uuid5(NAMESPACE, "socialcrawl-cohorts:" + material)) def call(session, method, path, key=None, body=None): headers = {} if key: headers["Idempotency-Key"] = key response = session.request(method, BASE + path, json=body, headers=headers) if response.status_code >= 400: sys.exit(f"{method} {path} -> {response.status_code}: {response.text[:400]}") return response.json()["data"] if response.text else None def main() -> None: parser = argparse.ArgumentParser() parser.add_argument("panel_csv") parser.add_argument("--name", required=True) parser.add_argument("--keywords", nargs="+", required=True) parser.add_argument("--days", type=int, default=30) parser.add_argument("--max-pages", type=int, default=3) parser.add_argument("--max-credits", type=int, default=None) parser.add_argument("--api-key", default=None) args = parser.parse_args() import os api_key = args.api_key or os.environ["SOCIALCRAWL_API_KEY"] session = requests.Session() session.headers["x-api-key"] = api_key with open(args.panel_csv, newline="", encoding="utf-8") as handle: panel = list(csv.DictReader(handle)) # Dedupe with the SAME normalization the API applies (NFKC, strip one # leading @, casefold). Two different external_ids claiming one identity # is a hard 409 COHORT_IDENTITY_CONFLICT whether the rows share a chunk # or not, so fail fast here instead of dying mid-upload. def identity_key(row): handle = unicodedata.normalize("NFKC", row["handle"]).strip() if handle.startswith("@"): handle = handle[1:].strip() return (row["platform"].strip().casefold(), handle.casefold()) seen, deduped = {}, [] for row in panel: key = identity_key(row) if key not in seen: seen[key] = row.get("external_id") deduped.append(row) elif (row.get("external_id") or None) != (seen[key] or None): sys.exit( f"conflict: {row['platform']}/{row['handle']} appears under " f"two external_ids ({seen[key]!r} and {row.get('external_id')!r})" ) if len(deduped) < len(panel): print(f"note: merged {len(panel) - len(deduped)} duplicate rows") panel = deduped ceiling = credit_ceiling(panel, args.max_pages) max_credits = args.max_credits or ceiling if max_credits < ceiling: sys.exit(f"max_credits {max_credits} is below the ceiling {ceiling}; submission would 400") print(f"panel: {len(panel)} members, worst-case ceiling {ceiling} credits") # 1. Create (or replay) the cohort. Same name + retention -> same cohort. cohort = call(session, "POST", "/v1/cohorts", key=idem_key("create", args.name, "30"), body={"name": args.name, "retention_days": 30}) cohort_id = cohort["id"] print(f"cohort: {cohort_id}") # 2. Upload in chunks. The key includes the chunk content, so an edited # panel gets fresh keys while an identical re-run replays. for index in range(0, len(panel), CHUNK): chunk = [ {"external_id": row["external_id"], "platform": row["platform"], "handle": row["handle"]} for row in panel[index : index + CHUNK] ] digest = hashlib.sha256( json.dumps(chunk, sort_keys=True).encode() ).hexdigest() result = call(session, "PUT", f"/v1/cohorts/{cohort_id}/members", key=idem_key("members", cohort_id, str(index), digest), body={"members": chunk}) print(f" chunk {index // CHUNK + 1}: member_count={result['member_count']}") # 3. Submit (or replay) the query. Keyed on the full spec plus the window # date, so tomorrow's run is a new query and today's re-run is not. date_from = ( datetime.now(timezone.utc) - timedelta(days=args.days) ).replace(hour=0, minute=0, second=0, microsecond=0) spec = { # Sorted so a re-run with the same keywords in a different order # replays instead of submitting (and charging) a second query. "keywords": sorted(set(args.keywords)), "date_from": date_from.isoformat().replace("+00:00", "Z"), "max_pages_per_identity": args.max_pages, "max_items_per_identity": 100, "max_credits": max_credits, } query = call(session, "POST", f"/v1/cohorts/{cohort_id}/queries", key=idem_key("query", cohort_id, json.dumps(spec, sort_keys=True)), body=spec) query_id = query["query_id"] print(f"query: {query_id} shards={query['shard_count']} " f"reserved={query['reserved_credits']}") # 4. Poll with backoff until terminal. delay = 5 while True: status = call(session, "GET", f"/v1/cohort-queries/{query_id}") progress = status["progress"] print(f" {status['status']}: members " f"{progress['members_completed']}/{progress['members_total']}, " f"shards {progress['shards_completed']}/{progress['shards_total']}") if status["status"] in ("succeeded", "failed", "cancelled", "expired"): break time.sleep(delay) delay = min(delay * 2, 30) print(f"billing: charged={status['actual_credits']} " f"refunded={status['refunded_credits']} " f"reserved={status['reserved_credits']}") if status["status"] != "succeeded": sys.exit(f"query ended {status['status']}; results are only served " "for succeeded queries") # 5. Drain results by cursor; write matches + coverage keyed by external_id. matches, coverage, cursor = [], {}, None while True: path = f"/v1/cohort-queries/{query_id}/results?limit=500" if cursor: path += f"&cursor={requests.utils.quote(cursor)}" page = call(session, "GET", path) matches.extend(page["items"]) for record in page["coverage"]: coverage[record["member_id"]] = record cursor = page["next_cursor"] if not cursor: break with open("matches.ndjson", "w", encoding="utf-8") as out: for item in matches: out.write(json.dumps(item) + "\n") with open("coverage.csv", "w", newline="", encoding="utf-8") as out: fields = ["external_id", "platform", "status", "pages", "oldest_seen", "window_complete"] writer = csv.DictWriter(out, fieldnames=fields, extrasaction="ignore") writer.writeheader() for record in coverage.values(): writer.writerow(record) incomplete = sum( 1 for record in coverage.values() if not record["window_complete"] ) print(f"done: {len(matches)} matches, {len(coverage)} coverage rows " f"({incomplete} members short of the window boundary)") if status["status"] != "succeeded": sys.exit(1) if __name__ == "__main__": main() ``` ### Operational notes for a recurring pipeline - **The cohort is reusable.** Upload the panel once, then run as many queries against it as you like: one per brand, one per keyword set, one per reporting window. Each query bills only its own pages, and querying renews the retention clock. - **Refresh, do not rebuild.** A nightly re-push of the full panel is an upsert keyed on `external_id`. Changed handles update, unchanged rows do nothing, and the member count never inflates. - **Keep chunk order stable across retries.** The idempotency fingerprint preserves array order, so replaying the same members in a different order under the same key is a `409`. Build chunks from an ordered source such as a file or a sorted query, never from a set or an unordered map. - **Segment with cohorts, not filters.** One cohort per purchaser segment keeps each query's cost proportional to the segment, and keeps your segmentation logic on your side. - **Read `coverage` before concluding silence.** A member with `status: "complete"` and no matches genuinely did not post about you. A member with `window_complete: false` was not fully crawled. Those are different findings. ## Next steps - [Error handling](/docs/errors.md): Every cohort error code, with status, retry verdict, and refund behavior. - [Credits](/docs/credits.md): How reservations, metering, and refunds work across the API. - [Pagination](/docs/pagination.md): The cursor contract that the results endpoint follows. - [API reference](/docs/api-reference.md): The full OpenAPI surface, including every cohort route.