Streaming (SSE)

Some calls fan out across many sources and take tens of seconds to complete. Those endpoints can answer as a Server-Sent Events stream instead of a single JSON envelope: you get the first frame in well under a second and each result the moment it settles, rather than a spinner until the slowest leg returns.

Streaming changes the wire format, not the price. The same work is done and the same credits are settled either way.

Which endpoints stream

Every other endpoint ignores the header and returns the ordinary JSON envelope. Nothing breaks if you send Accept: text/event-stream to an endpoint that does not stream.

Negotiating a stream

Add the header. Use curl -N so curl does not buffer the output:

curl -N 'https://www.socialcrawl.dev/v1/search/everywhere?query=nvidia+earnings' \
  -H 'x-api-key: sc_...' \
  -H 'Accept: text/event-stream'

The response carries content-type: text/event-stream; charset=utf-8, cache-control: no-cache, no-transform, and x-accel-buffering: no so intermediaries do not hold frames back.

If your HTTP client needs to be told not to buffer, this is where to configure it — a client that waits for the response to complete before handing you the body turns a stream back into a slow envelope.

Frame format

Every frame is one SSE data: line followed by a blank line:

data: {"type":"meta","request_id":"req-abc123","query":"nvidia earnings", ...}

data: {"type":"items","source":"youtube","items":[...],"duration_ms":1234}

There are no named SSE event: fields. Everything is discriminated by the type key inside the JSON, so a client can parse each data: payload and switch on type.

Two chunk vocabularies

The chunk types depend on which family you called.

Meta-search — search/everywhere and search/news

Planning and fan-out telemetry interleaved with results: meta, source_started, source_pruned, items, source_failed, plan_refined, ranked_partial, ranked_final, comments_enriched, transcript_enriched, clusters, warning, and a terminal done (or error then done). The universal search reference documents each one with a real payload and the ordering guarantees.

Prism composites and batch — everything else in the table

A smaller vocabulary:

result.key is a per-endpoint label rather than an index — stat for post-stats rows, item for comment-lookup, profile for profiles, transcript for transcripts, comments for prism/comments, and a final legs frame carrying the leg array on the endpoints that report one.

Terminal guarantees

• done is always last. If the handler forgets to emit one, the stream appends it before closing, so a client can always wait for done rather than for the socket to close.
• An error is a frame, not a dropped connection. A failure mid-run emits error and then done. The done summary tells you whether the call was refunded.
• Prism streams have a hard 55-second server-side ceiling. If the fan-out neither finishes nor settles within it, the stream refunds, emits error + done, and closes rather than holding your connection open indefinitely. Each composite's own per-leg deadlines normally settle long before this fires, and batch endpoints additionally cut off at 45 seconds with deferred rows.

Billing on a stream

There is no envelope, so there is no credits_used field and no credits_remaining to read at the end of the body. The terminal done event carries the figures instead:

• Meta-search: summary.credits_used, plus refunded and partial_failure.
• Prism batch: the nine-field batch summary, including credits_charged and credits_refunded. It is the identical object the sync JSON response returns, so the two can never disagree.

If you disconnect early — a closed tab, a refresh, a dropped network — the server sees the cancellation, aborts the in-flight upstream and LLM work so it stops spending, and refunds the portion that was never delivered. You are not charged for results you did not receive.

Streams are never cached. A streaming request also cannot poison a later sync request's cache: the two are keyed separately, so asking for a stream and then asking for JSON gives you a real second run rather than a replayed stream.

Concurrency and rate limits

A stream counts as one request against the 600 requests-per-minute budget, the same as any other call. Its concurrency slot is released once the response begins, not when the last frame lands, so a long-running stream does not hold one of your 50 concurrent slots open for its whole duration. See Rate limits.

When to stream

• Rendering results in a UI. search/everywhere takes roughly 12 to 30 seconds to complete but emits its first frame in about 300 ms. Streaming is the difference between a spinner and a list that fills in.
• Large batches. Processing rows as they settle beats blocking on the slowest item, and it means a deferred tail costs you nothing in wall-clock time.
• Long composites. prism/answers streams each engine's answer as it lands, which is why it has no sync form at all.

Stay on the sync JSON envelope for scheduled jobs, workflow tools, and anything that just wants the finished object — it is simpler to parse and gives you the credit fields directly.

Where to go next

• Universal search for the full meta-search chunk reference with real payloads.
• Batch endpoints for per-row streaming and the deferred deadline.
• Prism composites for which composites fan out and what they cost.
• Response schema for the sync envelope the stream replaces.