# Streaming (SSE) (/docs/streaming)

Which endpoints answer with Server-Sent Events, how to negotiate a stream, the frame format and the two chunk vocabularies, and how billing settles on a stream you disconnect early.

The endpoints that fan out across many sources can answer as a Server-Sent Events stream instead of a single JSON envelope. You get the first frame in well under a second and each result the moment it settles, rather than a spinner until the slowest leg returns.

Streaming changes the wire format, not the price. The same work is done and the same credits are settled either way.

## Which endpoints stream

| Endpoint                        | How it streams                                          |
| ------------------------------- | ------------------------------------------------------- |
| `GET /v1/search/everywhere`     | `Accept: text/event-stream`                             |
| `GET /v1/search/news`           | `Accept: text/event-stream`                             |
| `GET /v1/reddit/omni-search`    | `Accept: text/event-stream`                             |
| `GET /v1/prism/comments`        | `Accept: text/event-stream`                             |
| `GET /v1/prism/app-reviews`     | `Accept: text/event-stream`                             |
| `GET /v1/prism/video-intel`     | Automatically, whenever `include` contains `transcript` |
| `GET /v1/prism/answers`         | **Always.** SSE is its only response format             |
| `POST /v1/prism/post-stats`     | `Accept: text/event-stream`                             |
| `POST /v1/prism/comment-lookup` | `Accept: text/event-stream`                             |
| `POST /v1/prism/profiles`       | `Accept: text/event-stream`                             |
| `POST /v1/youtube/transcripts`  | `Accept: text/event-stream`                             |

Every other endpoint ignores the header and returns the ordinary JSON envelope. Nothing breaks if you send `Accept: text/event-stream` to an endpoint that does not stream.

## Negotiating a stream

Add the header. With curl, use `-N` so the output is not buffered:

Do not add `Idempotency-Key` to a GET stream. That combination returns an unbilled `400 INVALID_REQUEST` before cache, billing, or upstream dispatch because an interrupted GET stream has no complete replay body. POST batch streams keep their sync-equivalent idempotency behavior.

```bash title="cURL"
curl -N 'https://www.socialcrawl.dev/v1/search/everywhere?query=nvidia+earnings' \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Accept: text/event-stream"
```

```ts title="TypeScript"
const res = await fetch(
  "https://www.socialcrawl.dev/v1/search/everywhere?query=nvidia+earnings",
  {
    headers: {
      "x-api-key": process.env.SOCIALCRAWL_API_KEY!,
      Accept: "text/event-stream",
    },
  },
);

const reader = res.body!.pipeThrough(new TextDecoderStream()).getReader();
let buffer = "";

for (;;) {
  const { value, done } = await reader.read();
  if (done) break;

  buffer += value;
  const frames = buffer.split("\n\n");
  buffer = frames.pop() ?? "";

  for (const frame of frames) {
    const line = frame.trim();
    if (!line.startsWith("data:")) continue;

    const chunk = JSON.parse(line.slice(5));
    if (chunk.type === "done") console.log(chunk.summary);
  }
}
```

```python title="Python"
import json
import os

import requests

with requests.get(
    "https://www.socialcrawl.dev/v1/search/everywhere",
    params={"query": "nvidia earnings"},
    headers={
        "x-api-key": os.environ["SOCIALCRAWL_API_KEY"],
        "Accept": "text/event-stream",
    },
    stream=True,
) as res:
    for line in res.iter_lines(decode_unicode=True):
        if not line or not line.startswith("data:"):
            continue

        chunk = json.loads(line[5:])
        if chunk["type"] == "done":
            print(chunk["summary"])
```

The response carries `content-type: text/event-stream; charset=utf-8`, `cache-control: no-cache, no-transform`, and `x-accel-buffering: no`, so intermediaries do not hold frames back. If your HTTP client needs to be told not to buffer, this is where to configure it. A client that waits for the response to complete before handing you the body turns a stream back into a slow envelope.

## Frame format

Every frame is one SSE `data:` line followed by a blank line:

```text title="Stream"
data: {"type":"meta","request_id":"req-abc123","query":"nvidia earnings", ...}

data: {"type":"items","source":"youtube","items":[...],"duration_ms":1234}

```

There are no named SSE `event:` fields. Everything is discriminated by the `type` key inside the JSON, so a client can parse each `data:` payload and switch on `type`.

## Two chunk vocabularies

The chunk types depend on which family you called.

### Meta-search: search/everywhere and search/news

Planning and fan-out telemetry interleaved with results: `meta`, `source_started`, `source_pruned`, `items`, `source_failed`, `plan_refined`, `ranked_partial`, `ranked_final`, `comments_enriched`, `transcript_enriched`, `clusters`, `warning`, and a terminal `done` (or `error` then `done`). The [universal search reference](/docs/search/everywhere.md#streaming-sse) documents each one with a real payload and the ordering guarantees.

### Prism composites and batch: everything else in the table

A smaller vocabulary:

| `type`   | Shape                                                   | Meaning                                                      |
| -------- | ------------------------------------------------------- | ------------------------------------------------------------ |
| `leg`    | `{ endpoint, status, credits_used, latency_ms, error }` | One internal upstream call settled. Telemetry, not a result. |
| `result` | `{ key, value }`                                        | One block of the answer. `key` names what it is.             |
| `done`   | `{ summary }`                                           | Terminal. Carries coverage and the billing figures.          |
| `error`  | `{ code, message }`                                     | Something failed; a `done` always follows.                   |

`result.key` is a per-endpoint label rather than an index: `stat` for `post-stats` rows, `item` for `comment-lookup`, `profile` for `profiles`, `transcript` for `transcripts`, and `comments` for `prism/comments`. A final `legs` frame carries the leg array on the endpoints that report one.

## Terminal guarantees

- **`done` is always last.** If the handler forgets to emit one, the stream appends it before closing, so a client can always wait for `done` rather than for the socket to close.
- **An error is a frame, not a dropped connection.** A failure mid-run emits `error` and then `done`. The terminal summary reports committed billing state: a rejected immediate refund remains `refunded:false` with the retained charge while exact compensation is queued.
- **Prism streams have a hard 55-second server-side ceiling.** If the fan-out neither finishes nor settles within it, the stream attempts its refund, emits `error` then `done`, and closes rather than holding your connection open indefinitely. Each composite's own per-leg deadlines normally settle long before this fires, and [batch endpoints](/docs/batch.md) additionally cut off at 45 seconds with `deferred` rows.

## Billing on a stream

There is no envelope, so there is no `credits_used` field and no `credits_remaining` to read at the end of the body. The terminal `done` event carries the figures instead:

- **Meta-search**: `summary.credits_used`, plus `refunded` and `partial_failure`.
- **Prism batch**: the nine-field batch summary, including `credits_charged` and `credits_refunded`. It is the identical object the sync JSON response returns, so the two can never disagree.

Normal settlement and emergency recovery are serialized. If a timeout closes the stream while the executor is still unwinding, only one path can reduce the held charge; a late continuation exits instead of issuing a second refund.

  If you disconnect early, from a closed tab, a refresh, or a dropped network,
  the server sees the cancellation, aborts the in-flight upstream and model work
  so it stops spending, and refunds the portion that was never delivered. You
  are not charged for results you did not receive.

**Streams are never cached.** A streaming request also cannot poison a later sync request's cache, because the two are keyed separately. Asking for a stream and then asking for JSON gives you a real second run rather than a replayed stream.

## Concurrency and rate limits

A stream counts as one request against the 600 requests-per-minute budget, the same as any other call. Its concurrency slot is released once the response begins, not when the last frame lands, so a long-running stream does not hold one of your 50 concurrent slots open for its whole duration. See [Rate limits](/docs/rate-limits.md).

## When to stream

- **Rendering results in a UI.** `search/everywhere` takes roughly 12 to 30 seconds to complete but emits its first frame in about 300 ms. Streaming is the difference between a spinner and a list that fills in.
- **Large batches.** Processing rows as they settle beats blocking on the slowest item, and it means a `deferred` tail costs you nothing in wall-clock time.
- **Long composites.** `prism/answers` streams each engine's answer as it lands, which is why it has no sync form at all.

Stay on the sync JSON envelope for scheduled jobs, workflow tools, and anything that just wants the finished object. It is simpler to parse and gives you the credit fields directly.

## Next steps

- [Universal search](/docs/search/everywhere.md#streaming-sse): The full meta-search chunk reference with real payloads.
- [Batch endpoints](/docs/batch.md): Per-row streaming and the deferred deadline.
- [Prism composites](/docs/prism.md): Which composites fan out, and what they cost.
- [Response schema](/docs/response-schema.md): The sync envelope a stream replaces.