# Universal social search (/docs/search/everywhere) `GET /v1/search/everywhere` [#get-v1searcheverywhere] Send one query and get back a ranked, clustered, comment-enriched set of results pulled in parallel from up to 17 sources across 14 platforms (Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, Threads, Pinterest, LinkedIn, Rumble, plus Perplexity and Tavily web synthesis). An LLM plans the query, results are RRF-fused and LLM-reranked, near-duplicates are clustered, and every result carries the top comments from its source so you see what real people said, not just titles. Returns one JSON envelope by default, or a live SSE stream when you ask for one. ```bash curl 'https://www.socialcrawl.dev/v1/search/everywhere?query=apple+vision+pro+reviews' \ -H 'x-api-key: sc_...' ``` **Credit cost:** flat **20 credits** per call, regardless of how many sources fire or how many comments are enriched. This is the only endpoint that bypasses the per-source 1/5/10 tier ladder. See [Endpoint pricing](/docs/endpoint-pricing) for the canonical, registry-derived cost table. **Auth:** `x-api-key` header, same as every other endpoint. **Method:** `GET`, all inputs as query params. *** Request parameters [#request-parameters] | Param | Required | Type | Default | Description | | --------------- | -------- | ----------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | `query` | yes | string (1 to 512 chars) | none | The natural-language search query. A leading `#` puts the query into hashtag mode (see [Modes](#modes)). | | `lookback_days` | no | integer (>= 1) | `30` | Recency window in days. Mutually exclusive with `from_date` / `to_date`. Windows over 90 days degrade coverage on date-limited sources (see below). | | `from_date` | no | ISO `YYYY-MM-DD` | none | Lower bound of an explicit date window. Mutually exclusive with `lookback_days`. | | `to_date` | no | ISO `YYYY-MM-DD` | today | Upper bound of an explicit date window. Defaults to today when `from_date` is set alone. | | `sources` | no | CSV string | all | Allowlist of source names to fan out to (e.g. `reddit,youtube,hackernews`). Mutually exclusive with `exclude`. | | `exclude` | no | CSV string | none | Blocklist of source names to skip. Mutually exclusive with `sources`. | Unknown source names in `sources` / `exclude` are silently ignored (they are filtered against the known source list). There is no error and no discount, so a typo costs you a source at full price — the commonest one is passing `twitter` instead of `twitter-ai-search`, which drops X coverage from the sweep while still charging the flat 20 credits. Check `data.sources_called` (or the SSE `done` summary) against what you asked for. Passing both `sources` and `exclude` returns `400 INVALID_REQUEST`. For the opt-in `include_transcripts` flag, see [Transcript enrichment](#transcript-enrichment-opt-in). Sources [#sources] The default fan-out hits every source below. Use `sources` / `exclude` to narrow it. The three `*-hashtag` siblings only fire when the query is in [hashtag mode](#modes); allowlisting a base platform does not pull in its hashtag sibling (`sources=instagram` does not include `instagram-hashtag`). | Source name | Platform / kind | Comment-enriched | | ------------------- | --------------------------------------- | ---------------- | | `reddit` | Reddit | yes | | `twitter-ai-search` | X / Twitter (Grok-backed AI synthesis) | no | | `youtube` | YouTube | yes | | `tiktok` | TikTok | yes | | `instagram` | Instagram | yes | | `hackernews` | Hacker News | yes | | `polymarket` | Polymarket prediction markets | no | | `github` | GitHub | yes | | `threads` | Threads | no | | `pinterest` | Pinterest | no | | `perplexity` | Perplexity (web synthesis) | no | | `tavily` | Tavily (grounded web results) | no | | `linkedin` | LinkedIn | no | | `rumble` | Rumble | no | | `tiktok-hashtag` | TikTok hashtag search (hashtag mode) | yes | | `instagram-hashtag` | Instagram hashtag search (hashtag mode) | no | | `youtube-hashtag` | YouTube hashtag search (hashtag mode) | yes | That is **14 base platforms plus 3 hashtag siblings = up to 17 sources** on a single hashtag-mode call. *** Modes [#modes] There is no user-facing mode switch. Keyword and hashtag fan-out are decided automatically by the planner: * **Keyword mode (default).** A natural-language query fans out to the base keyword-search sources. The planner also decomposes the query into weighted subqueries and picks which sources each subquery targets. * **Hashtag mode.** The three hashtag siblings (`tiktok-hashtag`, `instagram-hashtag`, `youtube-hashtag`) are added to the fan-out automatically in either of two cases: 1. **Explicit tag.** The query is a single hashtag token, e.g. `query=%23thrifting`. The leading `#` is stripped before dispatch. 2. **Topic term.** For a multi-word topic query, the LLM planner extracts a topic term (e.g. `makeup tutorials` yields the tag `makeup`) and appends a hashtag-sibling subquery at half weight, so hashtag signal never dominates fusion on an ambiguous query. When the same TikTok / Instagram / YouTube post appears in both the keyword stream and the hashtag stream, fusion merges the two into one candidate with multi-source provenance and a higher fused score. If the LLM planner times out or fails, explicit `#tag` queries still route to hashtag mode deterministically; multi-word topic queries degrade to keyword-only. The `query` value is passed to the planner as-is. There are no special search operators (no `site:`, no boolean `AND`/`OR`, no quoting). The single input with special meaning is a leading `#`. *** What you get back (JSON) [#what-you-get-back-json] By default (`Accept: application/json`) you get one envelope. `data` carries: | Field | Type | Description | | ------------------- | ----------------------------- | ----------------------------------------------------------------------------------------------------------- | | `query` | string | The query as the planner saw it (`#` stripped for explicit-tag queries). | | `plan` | object | The resolved plan: `intent`, `freshness_mode`, `cluster_mode`, `subqueries[]`, `source_weights`, `notes[]`. | | `items` | array | The ranked, deduped candidates (up to 40). Each carries scoring plus its `source_items[]`. | | `items_by_source` | object | The same items grouped by source name, for per-platform rendering. | | `clusters` | array | Near-duplicate / same-story groupings over `items`. | | `sources_called` | array | Every source the pipeline attempted this request. | | `sources_failed` | object (`{ source: reason }`) | Sources that failed or were pruned, mapped to a reason string. | | `sources_succeeded` | array | Sources that returned without failing. | | `coverage` | number (0 to 1) | `sources_succeeded / sources_called`. Drives the partial-refund decision (see [Billing](#billing)). | | `partial_failure` | boolean | `true` when a partial-coverage refund fired this request. | A trimmed real response for `query=apple vision pro reviews` (envelope wrapped by the standard `{ success, data }`): ```json { "success": true, "data": { "query": "apple vision pro reviews", "plan": { "intent": "opinion", "freshness_mode": "balanced_recent", "cluster_mode": "debate", "subqueries": [ { "label": "reviews", "search_query": "apple vision pro review", "ranking_query": "hands-on apple vision pro review verdict", "sources": ["reddit", "youtube", "hackernews"], "weight": 1.0 } ], "source_weights": { "hackernews": 0.8, "youtube": 0.85, "reddit": 0.6 }, "notes": [] }, "items": [ { "candidate_id": "https://news.ycombinator.com/item?id=...", "source": "hackernews", "title": "Apple Vision Pro review: magic, until it's not", "url": "https://news.ycombinator.com/item?id=...", "rerank_score": 93, "final_score": 64.88, "sources": ["hackernews"], "cluster_id": "c1", "source_items": [ { "engagement": { "points": 412, "num_comments": 380 }, "media": { "thumbnail_url": null }, "metadata": { "top_comments": [ { "score": null, "excerpt": "This strikes me as the best review I've seen if you actually want to know what daily use is like...", "author": "some_real_user", "url": "https://news.ycombinator.com/item?id=...", "date": "2026-02-05T12:34:56Z" } ] } } ] } ], "items_by_source": { "hackernews": [], "youtube": [] }, "clusters": [ { "cluster_id": "c1", "title": "Vision Pro daily-use verdicts", "candidate_ids": ["..."], "representative_ids": ["..."], "sources": ["hackernews", "youtube"], "score": 0.71 } ], "sources_called": ["hackernews", "youtube", "reddit"], "sources_failed": { "reddit": "per-source timeout after 12000ms" }, "sources_succeeded": ["hackernews", "youtube"], "coverage": 0.6666666666666666, "partial_failure": false } } ``` Note in this real call `reddit` timed out (12s per-source budget), so it lands in `sources_failed`, `coverage` is `0.67`, and because that is still at or above the coverage floor the full 20 credits are charged (`partial_failure: false`). That is the normal, healthy shape: partial results are billable. *** Filter examples [#filter-examples] Each block is one call illustrating one input. **Narrow to specific sources** (`sources`): ```bash curl 'https://www.socialcrawl.dev/v1/search/everywhere?query=apple+vision+pro+reviews&sources=reddit,youtube,hackernews' \ -H 'x-api-key: sc_...' ``` Only `reddit`, `youtube`, and `hackernews` are fanned out; `sources_called` is exactly those three. **Exclude noisy sources** (`exclude`): ```bash curl 'https://www.socialcrawl.dev/v1/search/everywhere?query=nvidia+earnings&exclude=pinterest,polymarket' \ -H 'x-api-key: sc_...' ``` **Explicit date window** (`from_date` / `to_date`): ```bash curl 'https://www.socialcrawl.dev/v1/search/everywhere?query=world+cup&from_date=2026-06-01&to_date=2026-06-30' \ -H 'x-api-key: sc_...' ``` **Recency window** (`lookback_days`): ```bash curl 'https://www.socialcrawl.dev/v1/search/everywhere?query=nvidia+earnings&lookback_days=7' \ -H 'x-api-key: sc_...' ``` **Hashtag mode** (leading `#`, URL-encoded as `%23`): ```bash curl 'https://www.socialcrawl.dev/v1/search/everywhere?query=%23thrifting' \ -H 'x-api-key: sc_...' ``` In a real `#thrifting` call the plan intent resolves to `how_to`, and `sources_called` includes `tiktok-hashtag`, `instagram-hashtag`, and `youtube-hashtag` alongside the base sources. *** Real-people comments: `metadata.top_comments[]` [#real-people-comments-metadatatop_comments] For every result whose source exposes a comments endpoint, the pipeline automatically fetches the top-scoring comments and attaches them at `data.items[i].source_items[0].metadata.top_comments[]`. No second API call. | Source | Comments come from | | ------------ | --------------------------------------------------- | | `reddit` | `/v1/reddit/post/comments` | | `tiktok` | `/v1/tiktok/post/comments` | | `instagram` | `/v1/instagram/post/comments` | | `youtube` | `/v1/youtube/video/comments` | | `hackernews` | Algolia HN thread tree | | `github` | `/repos/{o}/{r}/issues/{n}/comments?sort=reactions` | Sources without a comments endpoint (`twitter-ai-search`, `threads`, `pinterest`, `polymarket`, `perplexity`, `tavily`, `linkedin`, `rumble`) render no `top_comments`. Each `TopComment` has this shape: ```ts { score: number | null, // upvote / like / points count, source-specific excerpt: string, // up to 300 chars, HTML stripped author: string | null, // null when [deleted] / [removed] url: string | null, // direct comment URL when the source exposes one date: string | null // ISO timestamp when available } ``` Comments are sorted by `score` descending and capped at 5 per result, 300 characters per excerpt. *** Streaming (SSE) [#streaming-sse] Add `Accept: text/event-stream` and you get a live stream instead of one envelope. First byte lands in well under a second (typically \~300ms on the `meta` chunk); fast sources stream their items while slow ones are still running. ```bash curl -N 'https://www.socialcrawl.dev/v1/search/everywhere?query=nvidia+earnings' \ -H 'x-api-key: sc_...' \ -H 'Accept: text/event-stream' ``` Each event is one `data: \n\n` frame. The chunk types, discriminated by `type`: ``` { "type": "meta", "request_id": "...", "query": "...", "plan": {...}, "sources_planned": [...] } { "type": "source_started", "source": "reddit" } { "type": "source_pruned", "source": "github", "reason": "low_affinity", "affinity": 0.1 } { "type": "items", "source": "youtube", "items": [...], "duration_ms": 1234 } { "type": "source_failed", "source": "linkedin", "error": { "code": "...", "message": "upstream linkedin returned 401" } } { "type": "plan_refined", "plan": {...} } { "type": "ranked_partial", "scores": [{ "candidate_id": "...", "rerank_score": 88 }] } { "type": "ranked_final", "items": [...] } { "type": "comments_enriched", "candidate_id": "...", "source": "reddit", "comments": [{ "score": 4823, "excerpt": "...", "author": "...", "url": "...", "date": "..." }] } { "type": "transcript_enriched","candidate_id": "...", "source": "youtube", "transcript": {...} } { "type": "clusters", "clusters": [...] } { "type": "warning", "message": "lookback_days > 90 degrades coverage on some sources" } { "type": "done", "summary": {...} } { "type": "error", "code": "INTERNAL_ERROR", "message": "..." } ``` Order and guarantees, verified against a live `nvidia earnings` stream: * **`meta` is first**, emitted as soon as the request is parsed. It carries the deterministic plan and the full `sources_planned` list (all 17 sources in a hashtag-mode plan). * **`source_started` fires once per source** before that source's adapter runs. Pruned or failing sources still emit `source_started`, then a `source_pruned` or `source_failed`. * **`source_pruned`** fires when the fast planner judged a source irrelevant for the query (e.g. `polymarket` on a phone-launch query). Its upstream call is aborted so it never bills, and it appears in the final `sources_failed` map with a `pruned: low_affinity (...)` reason. * **`items` yield in completion order**, so fast sources land first. * **`plan_refined` is conditional**: emitted only when the LLM-refined plan differs from the deterministic one. No chunk if the LLM times out or returns the plan unchanged. * **`ranked_partial` streams repeatedly** as the reranker scores batches; the frontend can update row scores live. * **`comments_enriched` is per-candidate**: one fires for each result whose comments were fetched. Key on `candidate_id` to merge comments into the row already on screen. These always land before `done`, and may interleave with `ranked_partial` / `ranked_final`. * **`transcript_enriched`** fires only when [`include_transcripts=true`](#transcript-enrichment-opt-in). * **`clusters`** is emitted when the intent is clusterable. * **`done` is always last**, carrying the run summary (below). On a thrown error you get a terminal `error` followed by `done` with `refunded: true`. The `done` summary (a real one): ```json { "type": "done", "summary": { "sources_called": ["youtube", "hackernews", "tiktok", "reddit", "..."], "sources_failed": { "linkedin": "upstream linkedin returned 401", "github": "pruned: low_affinity (0.10)" }, "sources_succeeded": ["youtube", "hackernews", "tiktok", "reddit", "..."], "coverage": 0.6470588235294118, "total_items": 175, "credits_used": 20, "cached": false, "refunded": false, "partial_failure": false } } ``` Streaming responses are never cached, and a streaming request cannot poison a follow-up sync request's cache (the two are keyed separately). *** Transcript enrichment (opt-in) [#transcript-enrichment-opt-in] Set `include_transcripts=true` to attach spoken-word transcripts to the top video results. It is off by default and is not part of the standard OpenAPI parameter set, so treat it as experimental. * Accepts `true` or `1` (case-insensitive). Anything else is treated as `false`. * Only the top few video-shaped candidates by rank get a transcript; the rest are unaffected. * Transcripts populate at `source_items[0].metadata.transcript` (JSON) and stream as `transcript_enriched` chunks (SSE). * The endpoint stays flat 20 credits with the flag on. Transcript sources: `youtube` and `rumble` return timestamped segments; `tiktok` and `instagram` return a flat caption string with no timing. *** Billing [#billing] Flat **20 credits** per call, deducted up front. Two automatic refund paths: * **Zero-floor (full refund).** If every source fails or returns nothing so there are zero ranked items, the full 20 credits are refunded. In sync mode the call returns `502 ZERO_RESULTS`; in SSE mode the terminal `done` carries `refunded: true`. * **Partial-coverage (half refund).** If some results come back but a strict majority of the called sources failed (`coverage` below the floor), half the cost is refunded and `partial_failure` is `true`. Partial results above the floor are billed in full. The terminal `done` summary's `credits_used` and `refunded` fields are the source of truth for what you were actually charged. See [Credits](/docs/credits) and [Endpoint pricing](/docs/endpoint-pricing). *** Latency and failure modes [#latency-and-failure-modes] The pipeline runs plan, parallel fan-out (per-source timeout 12s), fuse and dedupe, comment enrichment (parallel with rerank), LLM rerank (15s budget, partial scores kept on timeout), and cluster. `twitter-ai-search` runs as a background source with a longer budget because Grok synthesis is slow; its result still lands before `done`. Streaming consumers see progress within \~1s of the first source finishing. Sync consumers wait for the whole pipeline, typically \~12 to 30s. * **A source times out or errors:** the pipeline continues; the source lands in `sources_failed`. SSE gets a `source_failed` chunk. * **A comment fetch fails for one result:** non-fatal; that row just has no `top_comments`. * **`lookback_days` over 90:** date-limited sources (`tiktok`, `instagram`, `threads`, `pinterest`, `reddit`) can under-cover the window; SSE emits a `warning`. * **Pinterest** may return empty even on broad queries (upstream vendor issue). *** Related [#related] * Architecture: [`docs/features/social-search/UNIVERSAL-SEARCH.md`](https://github.com/RidioCompany/SocialCrawl/blob/main/docs/features/social-search/UNIVERSAL-SEARCH.md) * Comment enrichment deep dive: [`docs/features/social-search/COMMENT-ENRICHMENT.md`](https://github.com/RidioCompany/SocialCrawl/blob/main/docs/features/social-search/COMMENT-ENRICHMENT.md) * Frontend SERP architecture: [`docs/features/social-search/FRONTEND-ARCHITECTURE.md`](https://github.com/RidioCompany/SocialCrawl/blob/main/docs/features/social-search/FRONTEND-ARCHITECTURE.md)