Universal social search

Send one query to GET /v1/search/everywhere and get back a ranked, clustered, comment-enriched set of results pulled in parallel from up to 17 sources across 14 platforms. An LLM plans the query, results are fused and reranked, near-duplicates are clustered, and every result carries the top comments from its source, so you see what real people said and not just titles.

Returns one JSON envelope by default, or a live SSE stream when you ask for one. Flat 20 credits per call.

curl "https://www.socialcrawl.dev/v1/search/everywhere?query=apple+vision+pro+reviews" \
  -H "x-api-key: YOUR_API_KEY"

How the fan-out works

The pipeline runs in five stages, and every one of them is visible in the response.

1. Plan. An LLM decomposes your query into weighted subqueries and picks which sources each subquery targets. A fast planner scores per-source relevance at the same time, and a source it judges irrelevant for the query is pruned, so its results are not ranked and it leaves the coverage denominator. The planner fires an abort on a pruned source, but a source that had already answered before the abort landed still reports its items, so treat a prune as "this source was not ranked" rather than as proof that no request was made.
2. Fan out. Every surviving source runs in parallel, each bounded by a 12-second per-source timeout. twitter-ai-search runs as a background source with a longer budget, because Grok synthesis is slow. Its result still lands before the run finishes.
3. Fuse and dedupe. Results are merged, and the same item found by two sources becomes one candidate with multi-source provenance and a higher fused score.
4. Enrich and rerank. Top comments are fetched in parallel with an LLM rerank pass that runs on a 15-second budget. Partial scores are kept on timeout.
5. Cluster. Near-duplicates and same-story groupings are collapsed into clusters, when the resolved intent is clusterable.

Sources

The default fan-out hits every source below. Use sources / exclude to narrow it.

The last three fire only in hashtag mode. That is 14 base platforms plus 3 hashtag siblings, so up to 17 sources on a single hashtag-mode call.

Platform shorthands. A bare platform name resolves to its whole group. twitter and x both reach twitter-ai-search, and youtube, instagram and tiktok each cover their -hashtag lane too, so exclude=youtube cannot leak YouTube content back in through youtube-hashtag. Exact lane names stay individually addressable. An unknown name in sources or exclude is a 400 that enumerates the valid names, never a silent drop.

Keyword mode and hashtag mode

There is no user-facing mode switch. The planner decides:

• Keyword mode (default). A natural-language query fans out to the base keyword-search sources.
• Hashtag mode. The three hashtag siblings are added automatically in either of two cases. Explicit tag: the query is a single hashtag token, for example query=%23thrifting, and the leading # is stripped before dispatch. Topic term: for a multi-word topic query the planner extracts a topic term (makeup tutorials yields the tag makeup) and appends a hashtag-sibling subquery at half weight, so hashtag signal never dominates fusion on an ambiguous query.

When the same TikTok, Instagram or YouTube post appears in both streams, fusion merges the two into one candidate.

If the planner times out or fails, explicit #tag queries still route to hashtag mode deterministically, and multi-word topic queries degrade to keyword-only.

The query value is passed to the planner as-is. There are no special search operators: no site:, no boolean AND/OR, no quoting. The single input with special meaning is a leading #.

Request parameters

from and to are accepted as aliases for from_date and to_date, so a date window carried over from /v1/search/news (which spells them from / to) is honoured here rather than silently dropped. See One date vocabulary across the four lanes. | sources | no | CSV string | all | Allowlist of source or platform names, for example reddit,youtube,hackernews. Mutually exclusive with exclude. | | exclude | no | CSV string | none | Blocklist of source or platform names to skip. Mutually exclusive with sources. | | include_transcripts | no | true / 1 | false | Opt-in spoken-word transcripts on the top 3 video results, so a video whose title never mentions your query but whose narration does still surfaces. |

Passing both sources and exclude returns 400 INVALID_REQUEST. Check data.sources_called (or the SSE done summary) against what you asked for.

Filter examples

# Narrow to specific sources
curl "https://www.socialcrawl.dev/v1/search/everywhere?query=apple+vision+pro+reviews&sources=reddit,youtube,hackernews" \
  -H "x-api-key: YOUR_API_KEY"

# Exclude noisy sources
curl "https://www.socialcrawl.dev/v1/search/everywhere?query=nvidia+earnings&exclude=pinterest,polymarket" \
  -H "x-api-key: YOUR_API_KEY"

# Explicit date window
curl "https://www.socialcrawl.dev/v1/search/everywhere?query=world+cup&from_date=2026-06-01&to_date=2026-06-30" \
  -H "x-api-key: YOUR_API_KEY"

# Recency window
curl "https://www.socialcrawl.dev/v1/search/everywhere?query=nvidia+earnings&lookback_days=7" \
  -H "x-api-key: YOUR_API_KEY"

# Hashtag mode (leading #, URL-encoded as %23)
curl "https://www.socialcrawl.dev/v1/search/everywhere?query=%23thrifting" \
  -H "x-api-key: YOUR_API_KEY"

In a real #thrifting call the plan intent resolves to how_to, and sources_called includes tiktok-hashtag, instagram-hashtag and youtube-hashtag alongside the base sources.

What you get back

By default (Accept: application/json) you get one envelope. data carries:

Real-people comments

For every result whose source exposes a comments endpoint, the pipeline fetches the top-scoring comments and attaches them at data.items[i].source_items[0].metadata.top_comments[]. No second API call.

Each TopComment has this shape:

{
  score: number | null,    // upvote / like / points count, source-specific
  excerpt: string,         // up to 300 chars, HTML stripped
  author: string | null,   // null when [deleted] / [removed]
  url: string | null,      // direct comment URL when the source exposes one
  date: string | null      // ISO timestamp when available
}

Comments are sorted by score descending and capped at 5 per result, 300 characters per excerpt. Sources without a comments endpoint (twitter-ai-search, threads, pinterest, polymarket, perplexity, tavily, linkedin, rumble) render no top_comments.

Streaming

Add Accept: text/event-stream and you get a live stream instead of one envelope. First byte lands in well under a second, typically around 300ms on the meta chunk, and fast sources stream their items while slow ones are still running.

curl -N "https://www.socialcrawl.dev/v1/search/everywhere?query=nvidia+earnings" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Accept: text/event-stream"

Each event is one data: <json>\n\n frame, discriminated by type. The order guarantees:

• meta is first, emitted as soon as the request is parsed. It carries the deterministic plan and the full sources_planned list.
• source_started fires once per source before that source's adapter runs. Pruned or failing sources still emit source_started, then a source_pruned or source_failed.
• source_pruned fires when the fast planner judged a source irrelevant for the query, and the source appears in sources_pruned (never in sources_failed, and never in the coverage denominator). The planner fires an abort on the source, but a source that had already answered before the abort landed still emits its items chunk, so treat source_pruned as "do not rank this source" rather than as proof that no request was made.
• items yield in completion order, so fast sources land first.
• plan_refined is conditional: emitted only when the LLM-refined plan differs from the deterministic one. No chunk if the LLM times out or returns the plan unchanged.
• ranked_partial streams repeatedly as the reranker scores batches. Each chunk carries only the scores that settled since the previous one, so merge by candidate_id rather than replacing your map. scored and total carry cumulative progress for a percentage indicator.
• comments_enriched is per-candidate. Key on candidate_id to merge comments into the row already on screen. These always land before done, and may interleave with ranked_partial / ranked_final.
• done is always last, carrying the run summary. On a thrown error you get a terminal error followed by done with refunded: true.

Streaming responses are never cached, and a streaming request cannot poison a follow-up sync request's cache. The two are keyed separately.

Billing

Flat 20 credits per call, deducted up front. Two automatic refund paths:

The terminal done summary's credits_used and refunded fields are the source of truth for what you were actually charged. See Credits and Endpoint pricing.

Latency and failure modes

Streaming consumers see progress within about a second of the first source finishing. Sync consumers wait for the whole pipeline, typically 12 to 30 seconds.

• A source times out or errors: the pipeline continues and the source lands in sources_failed. SSE gets a source_failed chunk.
• A comment fetch fails for one result: non-fatal. That row just has no top_comments.
• lookback_days over 90: date-limited sources (tiktok, instagram, threads, pinterest, reddit) can under-cover the window, and SSE emits a warning.
• Pinterest may return empty even on broad queries, because of an upstream vendor issue.

Next steps