GET /v1/search/everywhere

Send one query and get back a ranked, clustered, comment-enriched set of results pulled in parallel from up to 17 sources across 14 platforms (Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, Threads, Pinterest, LinkedIn, Rumble, plus Perplexity and Tavily web synthesis). An LLM plans the query, results are RRF-fused and LLM-reranked, near-duplicates are clustered, and every result carries the top comments from its source so you see what real people said, not just titles. Returns one JSON envelope by default, or a live SSE stream when you ask for one.

curl 'https://www.socialcrawl.dev/v1/search/everywhere?query=apple+vision+pro+reviews' \
  -H 'x-api-key: sc_...'

Credit cost: flat 20 credits per call, regardless of how many sources fire or how many comments are enriched. This is the only endpoint that bypasses the per-source 1/5/10 tier ladder. See Endpoint pricing for the canonical, registry-derived cost table.

Auth: x-api-key header, same as every other endpoint.

Method: GET, all inputs as query params.

Request parameters

Unknown source names in sources / exclude are silently ignored (they are filtered against the known source list). There is no error and no discount, so a typo costs you a source at full price — the commonest one is passing twitter instead of twitter-ai-search, which drops X coverage from the sweep while still charging the flat 20 credits. Check data.sources_called (or the SSE done summary) against what you asked for. Passing both sources and exclude returns 400 INVALID_REQUEST.

For the opt-in include_transcripts flag, see Transcript enrichment.

Sources

The default fan-out hits every source below. Use sources / exclude to narrow it. The three *-hashtag siblings only fire when the query is in hashtag mode; allowlisting a base platform does not pull in its hashtag sibling (sources=instagram does not include instagram-hashtag).

That is 14 base platforms plus 3 hashtag siblings = up to 17 sources on a single hashtag-mode call.

Modes

There is no user-facing mode switch. Keyword and hashtag fan-out are decided automatically by the planner:

• Keyword mode (default). A natural-language query fans out to the base keyword-search sources. The planner also decomposes the query into weighted subqueries and picks which sources each subquery targets.
• Hashtag mode. The three hashtag siblings (tiktok-hashtag, instagram-hashtag, youtube-hashtag) are added to the fan-out automatically in either of two cases:
  1. Explicit tag. The query is a single hashtag token, e.g. query=%23thrifting. The leading # is stripped before dispatch.
  2. Topic term. For a multi-word topic query, the LLM planner extracts a topic term (e.g. makeup tutorials yields the tag makeup) and appends a hashtag-sibling subquery at half weight, so hashtag signal never dominates fusion on an ambiguous query.

When the same TikTok / Instagram / YouTube post appears in both the keyword stream and the hashtag stream, fusion merges the two into one candidate with multi-source provenance and a higher fused score.

If the LLM planner times out or fails, explicit #tag queries still route to hashtag mode deterministically; multi-word topic queries degrade to keyword-only.

The query value is passed to the planner as-is. There are no special search operators (no site:, no boolean AND/OR, no quoting). The single input with special meaning is a leading #.

What you get back (JSON)

By default (Accept: application/json) you get one envelope. data carries:

A trimmed real response for query=apple vision pro reviews (envelope wrapped by the standard { success, data }):

{
  "success": true,
  "data": {
    "query": "apple vision pro reviews",
    "plan": {
      "intent": "opinion",
      "freshness_mode": "balanced_recent",
      "cluster_mode": "debate",
      "subqueries": [
        {
          "label": "reviews",
          "search_query": "apple vision pro review",
          "ranking_query": "hands-on apple vision pro review verdict",
          "sources": ["reddit", "youtube", "hackernews"],
          "weight": 1.0
        }
      ],
      "source_weights": { "hackernews": 0.8, "youtube": 0.85, "reddit": 0.6 },
      "notes": []
    },
    "items": [
      {
        "candidate_id": "https://news.ycombinator.com/item?id=...",
        "source": "hackernews",
        "title": "Apple Vision Pro review: magic, until it's not",
        "url": "https://news.ycombinator.com/item?id=...",
        "rerank_score": 93,
        "final_score": 64.88,
        "sources": ["hackernews"],
        "cluster_id": "c1",
        "source_items": [
          {
            "engagement": { "points": 412, "num_comments": 380 },
            "media": { "thumbnail_url": null },
            "metadata": {
              "top_comments": [
                {
                  "score": null,
                  "excerpt": "This strikes me as the best review I've seen if you actually want to know what daily use is like...",
                  "author": "some_real_user",
                  "url": "https://news.ycombinator.com/item?id=...",
                  "date": "2026-02-05T12:34:56Z"
                }
              ]
            }
          }
        ]
      }
    ],
    "items_by_source": { "hackernews": [], "youtube": [] },
    "clusters": [
      { "cluster_id": "c1", "title": "Vision Pro daily-use verdicts", "candidate_ids": ["..."], "representative_ids": ["..."], "sources": ["hackernews", "youtube"], "score": 0.71 }
    ],
    "sources_called": ["hackernews", "youtube", "reddit"],
    "sources_failed": { "reddit": "per-source timeout after 12000ms" },
    "sources_succeeded": ["hackernews", "youtube"],
    "coverage": 0.6666666666666666,
    "partial_failure": false
  }
}

Note in this real call reddit timed out (12s per-source budget), so it lands in sources_failed, coverage is 0.67, and because that is still at or above the coverage floor the full 20 credits are charged (partial_failure: false). That is the normal, healthy shape: partial results are billable.

Filter examples

Each block is one call illustrating one input.

Narrow to specific sources (sources):

curl 'https://www.socialcrawl.dev/v1/search/everywhere?query=apple+vision+pro+reviews&sources=reddit,youtube,hackernews' \
  -H 'x-api-key: sc_...'

Only reddit, youtube, and hackernews are fanned out; sources_called is exactly those three.

Exclude noisy sources (exclude):

curl 'https://www.socialcrawl.dev/v1/search/everywhere?query=nvidia+earnings&exclude=pinterest,polymarket' \
  -H 'x-api-key: sc_...'

Explicit date window (from_date / to_date):

curl 'https://www.socialcrawl.dev/v1/search/everywhere?query=world+cup&from_date=2026-06-01&to_date=2026-06-30' \
  -H 'x-api-key: sc_...'

Recency window (lookback_days):

curl 'https://www.socialcrawl.dev/v1/search/everywhere?query=nvidia+earnings&lookback_days=7' \
  -H 'x-api-key: sc_...'

Hashtag mode (leading #, URL-encoded as %23):

curl 'https://www.socialcrawl.dev/v1/search/everywhere?query=%23thrifting' \
  -H 'x-api-key: sc_...'

In a real #thrifting call the plan intent resolves to how_to, and sources_called includes tiktok-hashtag, instagram-hashtag, and youtube-hashtag alongside the base sources.

Real-people comments: metadata.top_comments[]

For every result whose source exposes a comments endpoint, the pipeline automatically fetches the top-scoring comments and attaches them at data.items[i].source_items[0].metadata.top_comments[]. No second API call.

Sources without a comments endpoint (twitter-ai-search, threads, pinterest, polymarket, perplexity, tavily, linkedin, rumble) render no top_comments.

Each TopComment has this shape:

{
  score: number | null,    // upvote / like / points count, source-specific
  excerpt: string,         // up to 300 chars, HTML stripped
  author: string | null,   // null when [deleted] / [removed]
  url: string | null,      // direct comment URL when the source exposes one
  date: string | null      // ISO timestamp when available
}

Comments are sorted by score descending and capped at 5 per result, 300 characters per excerpt.

Streaming (SSE)

Add Accept: text/event-stream and you get a live stream instead of one envelope. First byte lands in well under a second (typically ~300ms on the meta chunk); fast sources stream their items while slow ones are still running.

curl -N 'https://www.socialcrawl.dev/v1/search/everywhere?query=nvidia+earnings' \
  -H 'x-api-key: sc_...' \
  -H 'Accept: text/event-stream'

Each event is one data: <json>\n\n frame. The chunk types, discriminated by type:

{ "type": "meta",               "request_id": "...", "query": "...", "plan": {...}, "sources_planned": [...] }
{ "type": "source_started",     "source": "reddit" }
{ "type": "source_pruned",      "source": "github", "reason": "low_affinity", "affinity": 0.1 }
{ "type": "items",              "source": "youtube", "items": [...], "duration_ms": 1234 }
{ "type": "source_failed",      "source": "linkedin", "error": { "code": "...", "message": "upstream linkedin returned 401" } }
{ "type": "plan_refined",       "plan": {...} }
{ "type": "ranked_partial",     "scores": [{ "candidate_id": "...", "rerank_score": 88 }] }
{ "type": "ranked_final",       "items": [...] }
{ "type": "comments_enriched",  "candidate_id": "...", "source": "reddit", "comments": [{ "score": 4823, "excerpt": "...", "author": "...", "url": "...", "date": "..." }] }
{ "type": "transcript_enriched","candidate_id": "...", "source": "youtube", "transcript": {...} }
{ "type": "clusters",           "clusters": [...] }
{ "type": "warning",            "message": "lookback_days > 90 degrades coverage on some sources" }
{ "type": "done",               "summary": {...} }
{ "type": "error",              "code": "INTERNAL_ERROR", "message": "..." }

Order and guarantees, verified against a live nvidia earnings stream:

• meta is first, emitted as soon as the request is parsed. It carries the deterministic plan and the full sources_planned list (all 17 sources in a hashtag-mode plan).
• source_started fires once per source before that source's adapter runs. Pruned or failing sources still emit source_started, then a source_pruned or source_failed.
• source_pruned fires when the fast planner judged a source irrelevant for the query (e.g. polymarket on a phone-launch query). Its upstream call is aborted so it never bills, and it appears in the final sources_failed map with a pruned: low_affinity (...) reason.
• items yield in completion order, so fast sources land first.
• plan_refined is conditional: emitted only when the LLM-refined plan differs from the deterministic one. No chunk if the LLM times out or returns the plan unchanged.
• ranked_partial streams repeatedly as the reranker scores batches; the frontend can update row scores live.
• comments_enriched is per-candidate: one fires for each result whose comments were fetched. Key on candidate_id to merge comments into the row already on screen. These always land before done, and may interleave with ranked_partial / ranked_final.
• transcript_enriched fires only when include_transcripts=true.
• clusters is emitted when the intent is clusterable.
• done is always last, carrying the run summary (below). On a thrown error you get a terminal error followed by done with refunded: true.

The done summary (a real one):

{
  "type": "done",
  "summary": {
    "sources_called": ["youtube", "hackernews", "tiktok", "reddit", "..."],
    "sources_failed": {
      "linkedin": "upstream linkedin returned 401",
      "github": "pruned: low_affinity (0.10)"
    },
    "sources_succeeded": ["youtube", "hackernews", "tiktok", "reddit", "..."],
    "coverage": 0.6470588235294118,
    "total_items": 175,
    "credits_used": 20,
    "cached": false,
    "refunded": false,
    "partial_failure": false
  }
}

Streaming responses are never cached, and a streaming request cannot poison a follow-up sync request's cache (the two are keyed separately).

Transcript enrichment (opt-in)

Set include_transcripts=true to attach spoken-word transcripts to the top video results. It is off by default and is not part of the standard OpenAPI parameter set, so treat it as experimental.

• Accepts true or 1 (case-insensitive). Anything else is treated as false.
• Only the top few video-shaped candidates by rank get a transcript; the rest are unaffected.
• Transcripts populate at source_items[0].metadata.transcript (JSON) and stream as transcript_enriched chunks (SSE).
• The endpoint stays flat 20 credits with the flag on.

Transcript sources: youtube and rumble return timestamped segments; tiktok and instagram return a flat caption string with no timing.

Billing

Flat 20 credits per call, deducted up front. Two automatic refund paths:

• Zero-floor (full refund). If every source fails or returns nothing so there are zero ranked items, the full 20 credits are refunded. In sync mode the call returns 502 ZERO_RESULTS; in SSE mode the terminal done carries refunded: true.
• Partial-coverage (half refund). If some results come back but a strict majority of the called sources failed (coverage below the floor), half the cost is refunded and partial_failure is true. Partial results above the floor are billed in full.

The terminal done summary's credits_used and refunded fields are the source of truth for what you were actually charged. See Credits and Endpoint pricing.

Latency and failure modes

The pipeline runs plan, parallel fan-out (per-source timeout 12s), fuse and dedupe, comment enrichment (parallel with rerank), LLM rerank (15s budget, partial scores kept on timeout), and cluster. twitter-ai-search runs as a background source with a longer budget because Grok synthesis is slow; its result still lands before done. Streaming consumers see progress within ~1s of the first source finishing. Sync consumers wait for the whole pipeline, typically ~12 to 30s.

• A source times out or errors: the pipeline continues; the source lands in sources_failed. SSE gets a source_failed chunk.
• A comment fetch fails for one result: non-fatal; that row just has no top_comments.
• lookback_days over 90: date-limited sources (tiktok, instagram, threads, pinterest, reddit) can under-cover the window; SSE emits a warning.
• Pinterest may return empty even on broad queries (upstream vendor issue).

Related

• Architecture: docs/features/social-search/UNIVERSAL-SEARCH.md
• Comment enrichment deep dive: docs/features/social-search/COMMENT-ENRICHMENT.md
• Frontend SERP architecture: docs/features/social-search/FRONTEND-ARCHITECTURE.md