# Universal social search (/docs/search/everywhere) One endpoint that fans out across 15 social platforms (up to 18 sources) in parallel, then ranks, clusters, and comment-enriches the merged results. JSON or SSE. Send one query to `GET /v1/search/everywhere` and get back a ranked, clustered, comment-enriched set of results pulled in parallel from up to 18 sources across 15 platforms. An LLM plans the query, results are fused and reranked, near-duplicates are clustered, and every result carries the top comments from its source, so you see what real people said and not just titles. Returns one JSON envelope by default, or a live SSE stream when you ask for one. Flat **20 credits** per call. ```bash title="cURL" curl "https://www.socialcrawl.dev/v1/search/everywhere?query=apple+vision+pro+reviews" \ -H "x-api-key: YOUR_API_KEY" ``` This is the only endpoint that bypasses the per-source 1/5/10 tier ladder. The 20 credits are flat, however many sources fire and however many comments are enriched. ## How the fan-out works The pipeline runs in five stages, and every one of them is visible in the response. 1. **Plan.** An LLM decomposes your query into weighted subqueries and picks which sources each subquery targets. A fast planner scores per-source relevance at the same time, and a source it judges irrelevant for the query is **pruned**, so its results are not ranked and it leaves the coverage denominator. The planner fires an abort on a pruned source, but a source that had already answered before the abort landed still reports its `items`, so treat a prune as "this source was not ranked" rather than as proof that no request was made. When you name sources with `sources`, pruning never removes all of them: if every source you asked for scores below the floor, the highest-scoring one is still searched. 2. **Fan out.** Every surviving source runs in parallel, each bounded by a 12-second per-source timeout. `twitter-ai-search` runs as a background source with a longer budget, because Grok synthesis is slow. Its result still lands before the run finishes. 3. **Fuse and dedupe.** Results are merged, and the same item found by two sources becomes one candidate with multi-source provenance and a higher fused score. 4. **Enrich and rerank.** Top comments are fetched in parallel with an LLM rerank pass that runs on a 15-second budget. Partial scores are kept on timeout. Each result's stance toward your query is judged beside the rerank, and its relevance score comes from the rerank's own answers. 5. **Cluster.** Near-duplicates and same-story groupings are collapsed into `clusters`, when the resolved intent is clusterable. ### Sources The default fan-out hits every source below except `naver`, which joins when the query contains Hangul. Use `sources` / `exclude` to narrow it. | Source name | Platform or kind | Comment-enriched | | ------------------- | --------------------------------------- | ---------------- | | `reddit` | Reddit | yes | | `twitter-ai-search` | X / Twitter (Grok-backed AI synthesis) | no | | `youtube` | YouTube | yes | | `tiktok` | TikTok | yes | | `instagram` | Instagram | yes | | `hackernews` | Hacker News | yes | | `polymarket` | Polymarket prediction markets | no | | `github` | GitHub | yes | | `threads` | Threads | no | | `pinterest` | Pinterest | no | | `perplexity` | Perplexity (web synthesis) | no | | `tavily` | Tavily (grounded web results) | no | | `linkedin` | LinkedIn | no | | `rumble` | Rumble | no | | `tiktok-hashtag` | TikTok hashtag search (hashtag mode) | yes | | `instagram-hashtag` | Instagram hashtag search (hashtag mode) | no | | `youtube-hashtag` | YouTube hashtag search (hashtag mode) | yes | | `naver` | Naver blog, cafe and KnowledgeiN (Korean queries) | no | The three hashtag lanes fire only in hashtag mode. `naver` fires when the query contains Hangul, or on any query when you name it in `sources`. That is 15 base platforms plus 3 hashtag siblings, so up to 18 sources on one call. **Korean queries.** `naver` searches Naver blog posts, cafe communities and KnowledgeiN questions in one lane. Each Naver row carries its corpus in `metadata.corpus`. Blog rows have the posting day; KnowledgeiN rows have an estimated date, so their `date_confidence` is `low`; cafe rows have no date. **Dates.** Check each row's `date_confidence` before you filter on `published_at`. `low` means the date is an estimate: a KnowledgeiN estimate, or a YouTube row dated from a "3 years ago" or "2 months ago" label. **X on the JSON path.** `twitter-ai-search` writes a summary of X posts, which takes 15 to 45 seconds. On the JSON path it has 11 seconds, so it usually lands in `sources_failed` as a timeout. Use the streaming path for the X summary, or `/v1/twitter/search/tweets` for X posts. **Platform shorthands.** A bare platform name resolves to its whole group. `twitter` and `x` both reach `twitter-ai-search`, and `youtube`, `instagram` and `tiktok` each cover their `-hashtag` lane too, so `exclude=youtube` cannot leak YouTube content back in through `youtube-hashtag`. Exact lane names stay individually addressable. An unknown name in `sources` or `exclude` is a `400` that enumerates the valid names, never a silent drop. ### Keyword mode and hashtag mode There is no user-facing mode switch. The planner decides: - **Keyword mode (default).** A natural-language query fans out to the base keyword-search sources. - **Hashtag mode.** The three hashtag siblings are added automatically in either of two cases. **Explicit tag:** the query is a single hashtag token, for example `query=%23thrifting`, and the leading `#` is stripped before dispatch. **Topic term:** for a multi-word topic query the planner extracts a topic term (`makeup tutorials` yields the tag `makeup`) and appends a hashtag-sibling subquery at half weight, so hashtag signal never dominates fusion on an ambiguous query. When the same TikTok, Instagram or YouTube post appears in both streams, fusion merges the two into one candidate. If the planner times out or fails, explicit `#tag` queries still route to hashtag mode deterministically, and multi-word topic queries degrade to keyword-only. The `query` value is passed to the planner as-is. There are no special search operators: no `site:`, no boolean `AND`/`OR`, no quoting. The single input with special meaning is a leading `#`. ## Request parameters | Param | Required | Type | Default | Description | | --------------- | -------- | ----------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `query` | yes | string (1 to 512 chars) | none | The natural-language search query. A leading `#` puts the query into hashtag mode. | | `lookback_days` | no | integer (minimum 1) | `30` | Recency window in days. Zero or negative is rejected before billing; there is no upper bound, because this is a ranking signal rather than a hard filter and a very wide window is a degraded answer rather than an invalid request. Mutually exclusive with `from_date` / `to_date`. It sets the freshness curve and is forwarded to the sources whose upstream accepts date bounds, so an older item can still rank when nothing recent matches. Windows over 90 days degrade coverage on date-limited sources and emit a `warning` chunk. | | `from_date` | no | ISO `YYYY-MM-DD` | none | Lower bound of an explicit date window. Mutually exclusive with `lookback_days`. | | `to_date` | no | ISO `YYYY-MM-DD` | today | Upper bound of an explicit date window. Defaults to today when `from_date` is set alone. | | `sources` | no | CSV string | all | Allowlist of source or platform names, for example `reddit,youtube,hackernews`. Mutually exclusive with `exclude`. | | `exclude` | no | CSV string | none | Blocklist of source or platform names to skip. Mutually exclusive with `sources`. | | `include_transcripts` | no | `true` / `1` | `false` | Opt-in spoken-word transcripts on the top 3 video results, so a video whose title never mentions your query but whose narration does still surfaces. | | `judgments` | no | `on` \| `off` | `on` | `on` adds a stance (`computed.stance`) and a relevance score (`computed.relevance`) to every result, with `data.stance_split` and `data.relevance` counting them. Nothing is dropped or reordered and the price does not change. `off` returns the response without these blocks. | | `relevance` | no | `filter` | none | Drops the results the ranking step judged off-topic for your query and lists their ids in `data.relevance.dropped_ids`. A result that could not be judged is never dropped, and `items_by_source` is not filtered. Included in the flat 20 credits. | | `include` | no | CSV: `stance` | none | Kept for compatibility: stance is on by default now. Naming it gives the stance judgment priority when capacity is short. | `from` and `to` are accepted as aliases for `from_date` and `to_date`, so a date window carried over from `/v1/search/news` (which spells them `from` / `to`) is honoured here rather than silently dropped. See [One date vocabulary across the four lanes](/docs/search.md#one-date-vocabulary-across-the-four-lanes). Passing both `sources` and `exclude` returns `400 INVALID_REQUEST`. Check `data.sources_called` (or the SSE `done` summary) against what you asked for. ## Filter examples ```bash title="cURL" # Narrow to specific sources curl "https://www.socialcrawl.dev/v1/search/everywhere?query=apple+vision+pro+reviews&sources=reddit,youtube,hackernews" \ -H "x-api-key: YOUR_API_KEY" # Exclude noisy sources curl "https://www.socialcrawl.dev/v1/search/everywhere?query=nvidia+earnings&exclude=pinterest,polymarket" \ -H "x-api-key: YOUR_API_KEY" # Explicit date window curl "https://www.socialcrawl.dev/v1/search/everywhere?query=world+cup&from_date=2026-06-01&to_date=2026-06-30" \ -H "x-api-key: YOUR_API_KEY" # Recency window curl "https://www.socialcrawl.dev/v1/search/everywhere?query=nvidia+earnings&lookback_days=7" \ -H "x-api-key: YOUR_API_KEY" # Hashtag mode (leading #, URL-encoded as %23) curl "https://www.socialcrawl.dev/v1/search/everywhere?query=%23thrifting" \ -H "x-api-key: YOUR_API_KEY" ``` In a real `#thrifting` call the plan intent resolves to `how_to`, and `sources_called` includes `tiktok-hashtag`, `instagram-hashtag` and `youtube-hashtag` alongside the base sources. ## What you get back By default (`Accept: application/json`) you get one envelope. `data` carries: | Field | Type | Description | | ------------------- | ----------------------------- | ----------------------------------------------------------------------------------------------------------- | | `query` | string | The query as the planner saw it (`#` stripped for explicit-tag queries). | | `plan` | object | The resolved plan: `intent`, `freshness_mode`, `cluster_mode`, `subqueries[]`, `source_weights`, `notes[]`. | | `items` | array | The ranked, deduped candidates (up to 40). Each carries scoring plus its `source_items[]`. | | `items_by_source` | object | The same items grouped by source name, for per-platform rendering. | | `clusters` | array | Near-duplicate and same-story groupings over `items`. | | `sources_called` | array | Every source the pipeline attempted this request. | | `sources_failed` | object (`{ source: reason }`) | Sources that failed or were pruned, mapped to a reason string. | | `sources_succeeded` | array | Sources that returned without failing. | | `coverage` | number (0 to 1) | `sources_succeeded / sources_called`. Drives the partial-refund decision. | | `partial_failure` | boolean | `true` when a partial-coverage refund fired this request. | | `stance_split` | object | Stance counts and shares over the results, overall and per platform, with the ids behind every count. Absent with `judgments=off`. | | `relevance` | object | How many results were judged and how many are off-topic (`off_topic`, `off_topic_ids`); with `relevance=filter`, the `dropped_ids`. Absent with `judgments=off`. | | `_warnings` | array | Present only when something needs saying, such as `stance_partial`, `stance_unavailable`, `stance_skipped`, `relevance_partial` or `relevance_unavailable`. | ### Stance and relevance on every result Each result in `items` carries two judgments by default, free: ```json title="JSON" "computed": { "stance": { "label": "neutral", "probabilities": { "positive": 0.01, "negative": 0, "neutral": 0.92, "no_opinion": 0.07 }, "confidence": 0.9 }, "relevance": { "score": 0.93, "on_topic": true, "entity": 0.97 } } ``` `stance.label` is what the post, with any replies shown, says about your query: `positive`, `negative`, `neutral`, `no_opinion`, or `null` when the judgment is under the confidence floor, which counts as uncertain in `data.stance_split`, never as a stance. `relevance.score` runs from 0 to 1, and `on_topic: false` marks exactly the results `relevance=filter` would drop. Both are judged inside the request under a deadline that runs beside the ranking step, so they add no wait; a result not judged in time is `null` for the life of the cached response. [Labels](/docs/labels.md#how-we-measure) states how stance was measured. ### Real-people comments For every result whose source exposes a comments endpoint, the pipeline fetches the top-scoring comments and attaches them at `data.items[i].source_items[0].metadata.top_comments[]`. No second API call. Each `TopComment` has this shape: ```ts title="TypeScript" { score: number | null, // upvote / like / points count, source-specific excerpt: string, // up to 300 chars, HTML stripped author: string | null, // null when [deleted] / [removed] url: string | null, // direct comment URL when the source exposes one date: string | null // ISO timestamp when available } ``` Comments are sorted by `score` descending and capped at 5 per result, 300 characters per excerpt. Sources without a comments endpoint (`twitter-ai-search`, `threads`, `pinterest`, `polymarket`, `perplexity`, `tavily`, `linkedin`, `rumble`, `naver`) render no `top_comments`. | Source | Comments come from | | ------------ | --------------------------------------------------- | | `reddit` | `/v1/reddit/post/comments` | | `tiktok` | `/v1/tiktok/post/comments` | | `instagram` | `/v1/instagram/post/comments` | | `youtube` | `/v1/youtube/video/comments` | | `hackernews` | Algolia HN thread tree | | `github` | `/repos/{o}/{r}/issues/{n}/comments?sort=reactions` | For `query=apple vision pro reviews`, wrapped by the standard `{ success, data }` envelope: ```json title="Response" { "success": true, "data": { "query": "apple vision pro reviews", "plan": { "intent": "opinion", "freshness_mode": "balanced_recent", "cluster_mode": "debate", "subqueries": [ { "label": "reviews", "search_query": "apple vision pro review", "ranking_query": "hands-on apple vision pro review verdict", "sources": ["reddit", "youtube", "hackernews"], "weight": 1.0 } ], "source_weights": { "hackernews": 0.8, "youtube": 0.85, "reddit": 0.6 }, "notes": [] }, "items": [ { "candidate_id": "https://news.ycombinator.com/item?id=...", "source": "hackernews", "title": "Apple Vision Pro review: magic, until it's not", "url": "https://news.ycombinator.com/item?id=...", "rerank_score": 93, "final_score": 64.88, "sources": ["hackernews"], "cluster_id": "c1", "source_items": [ { "engagement": { "points": 412, "num_comments": 380 }, "media": { "thumbnail_url": null }, "metadata": { "top_comments": [ { "score": null, "excerpt": "This strikes me as the best review I've seen if you actually want to know what daily use is like...", "author": "some_real_user", "url": "https://news.ycombinator.com/item?id=...", "date": "2026-02-05T12:34:56Z" } ] } } ] } ], "items_by_source": { "hackernews": [], "youtube": [] }, "clusters": [ { "cluster_id": "c1", "title": "Vision Pro daily-use verdicts", "candidate_ids": ["..."], "representative_ids": ["..."], "sources": ["hackernews", "youtube"], "score": 0.71 } ], "sources_called": ["hackernews", "youtube", "reddit"], "sources_failed": { "reddit": "per-source timeout after 12000ms" }, "sources_succeeded": ["hackernews", "youtube"], "coverage": 0.6666666666666666, "partial_failure": false } } ``` In this real call `reddit` timed out on the 12-second per-source budget, so it lands in `sources_failed`, `coverage` is `0.67`, and because that is still at or above the coverage floor the full 20 credits are charged (`partial_failure: false`). That is the normal, healthy shape: partial results are billable. ## Streaming Add `Accept: text/event-stream` and you get a live stream instead of one envelope. First byte lands in well under a second, typically around 300ms on the `meta` chunk, and fast sources stream their items while slow ones are still running. ```bash title="cURL" curl -N "https://www.socialcrawl.dev/v1/search/everywhere?query=nvidia+earnings" \ -H "x-api-key: YOUR_API_KEY" \ -H "Accept: text/event-stream" ``` Each event is one `data: \n\n` frame, discriminated by `type`. The order guarantees: - **`meta` is first**, emitted as soon as the request is parsed. It carries the deterministic plan and the full `sources_planned` list. - **`source_started` fires once per source** before that source's adapter runs. Pruned or failing sources still emit `source_started`, then a `source_pruned` or `source_failed`. - **`source_pruned`** fires when the fast planner judged a source irrelevant for the query, and the source appears in `sources_pruned` (never in `sources_failed`, and never in the coverage denominator). The planner fires an abort on the source, but a source that had already answered before the abort landed still emits its `items` chunk, so treat `source_pruned` as "do not rank this source" rather than as proof that no request was made. - **`items` yield in completion order**, so fast sources land first. - **`plan_refined` is conditional**: emitted only when the LLM-refined plan differs from the deterministic one. No chunk if the LLM times out or returns the plan unchanged. - **`ranked_partial` streams repeatedly** as the reranker scores batches. Each chunk carries only the scores that settled since the previous one, so merge by `candidate_id` rather than replacing your map. `scored` and `total` carry cumulative progress for a percentage indicator. - **`comments_enriched` is per-candidate.** Key on `candidate_id` to merge comments into the row already on screen. These always land before `done`, and may interleave with `ranked_partial` / `ranked_final`. - **`done` is always last**, carrying the run summary. On a thrown error you get a terminal `error` followed by `done` with `refunded: true`. Streaming responses are never cached, and a streaming request cannot poison a follow-up sync request's cache. The two are keyed separately. ```text title="Stream" { "type": "meta", "request_id": "...", "query": "...", "plan": {...}, "sources_planned": [...] } { "type": "source_started", "source": "reddit" } { "type": "source_pruned", "source": "github", "reason": "low_affinity", "affinity": 0.1 } { "type": "items", "source": "youtube", "items": [...], "duration_ms": 1234 } { "type": "source_failed", "source": "linkedin", "error": { "code": "...", "message": "upstream linkedin returned 401" } } { "type": "plan_refined", "plan": {...} } { "type": "ranked_partial", "scores": [{ "candidate_id": "...", "rerank_score": 88 }], "scored": 12, "total": 40 } { "type": "ranked_final", "items": [...] } { "type": "comments_enriched", "candidate_id": "...", "source": "reddit", "comments": [{ "score": 4823, "excerpt": "...", "author": "...", "url": "...", "date": "..." }] } { "type": "transcript_enriched","candidate_id": "...", "source": "youtube", "transcript": {...} } { "type": "clusters", "clusters": [...] } { "type": "warning", "message": "lookback_days > 90 degrades coverage on some sources" } { "type": "done", "summary": {...} } { "type": "error", "code": "INTERNAL_ERROR", "message": "..." } ``` `clusters` is emitted when the intent is clusterable. `transcript_enriched` fires only when `include_transcripts=true`. The items on `ranked_final` carry the same `computed.stance` and `computed.relevance` blocks as the JSON response; the page-level `stance_split` and `relevance` reports are JSON only. ```json title="JSON" { "type": "done", "summary": { "sources_called": ["youtube", "hackernews", "tiktok", "reddit", "..."], "sources_failed": { "linkedin": "upstream linkedin returned 401", "github": "pruned: low_affinity (0.10)" }, "sources_succeeded": ["youtube", "hackernews", "tiktok", "reddit", "..."], "coverage": 0.6470588235294118, "total_items": 175, "credits_used": 20, "cached": false, "refunded": false, "partial_failure": false } } ``` Set `include_transcripts=true` to attach spoken-word transcripts to the top video results. It is off by default. - Accepts `true` or `1`, case-insensitive. Anything else is treated as `false`. - Only the top few video-shaped candidates by rank get a transcript. The rest are unaffected. - Transcripts populate at `source_items[0].metadata.transcript` in JSON mode and stream as `transcript_enriched` chunks over SSE. - The endpoint stays flat 20 credits with the flag on. - `youtube` and `rumble` return timestamped segments. `tiktok` and `instagram` return a flat caption string with no timing. ## Billing Flat **20 credits** per call, deducted up front. Two automatic refund paths: | Situation | What you pay | | --------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | Normal run, including one where some sources failed but coverage stayed at or above the floor | 20 credits | | Some results, but a strict majority of the called sources failed | 10 credits (half refunded), and `partial_failure` is `true` | | Every source you scoped answered, and none had a result | 0 credits (fully refunded). Sync returns `200` with `items: []` and an empty `sources_failed`; SSE sends a terminal `done` with `refunded: true` | | No ranked items because a source failed | 0 credits (fully refunded). Sync returns `502 ZERO_RESULTS` with the failure in `sources_failed`; SSE sends a terminal `done` with `refunded: true` | Stance, relevance and `relevance=filter` are included in the 20 credits: a call costs the same with `judgments=off`. The terminal `done` summary's `credits_used` and `refunded` fields are the source of truth for what you were actually charged. See [Credits](/docs/credits.md) and [Endpoint pricing](/docs/endpoint-pricing.md). ## Latency and failure modes Streaming consumers see progress within about a second of the first source finishing. Sync consumers wait for the whole pipeline, typically 12 to 30 seconds. - **A source times out or errors:** the pipeline continues and the source lands in `sources_failed`. SSE gets a `source_failed` chunk. - **A comment fetch fails for one result:** non-fatal. That row just has no `top_comments`. - **`lookback_days` over 90:** date-limited sources (`tiktok`, `instagram`, `threads`, `pinterest`, `reddit`) can under-cover the window, and SSE emits a `warning`. - **Pinterest** may return empty even on broad queries, because of an upstream vendor issue. ## Next steps - [Universal news search](/docs/search/news.md): The same idea for news, planned into angles and localized per country. - [Streaming](/docs/streaming.md): Consuming SSE across the API. - [Prism](/docs/prism.md): Cross-platform composites for brand, creator, and commerce questions.