Streaming (SSE)

The endpoints that fan out across many sources can answer as a Server-Sent Events stream instead of a single JSON envelope. You get the first frame in well under a second and each result the moment it settles, rather than a spinner until the slowest leg returns.

Streaming changes the wire format, not the price. The same work is done and the same credits are settled either way.

Which endpoints stream

Every other endpoint ignores the header and returns the ordinary JSON envelope. Nothing breaks if you send Accept: text/event-stream to an endpoint that does not stream.

Negotiating a stream

Add the header. With curl, use -N so the output is not buffered:

Do not add Idempotency-Key to a GET stream. That combination returns an unbilled 400 INVALID_REQUEST before cache, billing, or upstream dispatch because an interrupted GET stream has no complete replay body. POST batch streams keep their sync-equivalent idempotency behavior.

curl -N 'https://www.socialcrawl.dev/v1/search/everywhere?query=nvidia+earnings' \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Accept: text/event-stream"

const res = await fetch(
  "https://www.socialcrawl.dev/v1/search/everywhere?query=nvidia+earnings",
  {
    headers: {
      "x-api-key": process.env.SOCIALCRAWL_API_KEY!,
      Accept: "text/event-stream",
    },
  },
);

const reader = res.body!.pipeThrough(new TextDecoderStream()).getReader();
let buffer = "";

for (;;) {
  const { value, done } = await reader.read();
  if (done) break;

  buffer += value;
  const frames = buffer.split("\n\n");
  buffer = frames.pop() ?? "";

  for (const frame of frames) {
    const line = frame.trim();
    if (!line.startsWith("data:")) continue;

    const chunk = JSON.parse(line.slice(5));
    if (chunk.type === "done") console.log(chunk.summary);
  }
}

import json
import os

import requests

with requests.get(
    "https://www.socialcrawl.dev/v1/search/everywhere",
    params={"query": "nvidia earnings"},
    headers={
        "x-api-key": os.environ["SOCIALCRAWL_API_KEY"],
        "Accept": "text/event-stream",
    },
    stream=True,
) as res:
    for line in res.iter_lines(decode_unicode=True):
        if not line or not line.startswith("data:"):
            continue

        chunk = json.loads(line[5:])
        if chunk["type"] == "done":
            print(chunk["summary"])

The response carries content-type: text/event-stream; charset=utf-8, cache-control: no-cache, no-transform, and x-accel-buffering: no, so intermediaries do not hold frames back. If your HTTP client needs to be told not to buffer, this is where to configure it. A client that waits for the response to complete before handing you the body turns a stream back into a slow envelope.

Frame format

Every frame is one SSE data: line followed by a blank line:

data: {"type":"meta","request_id":"req-abc123","query":"nvidia earnings", ...}

data: {"type":"items","source":"youtube","items":[...],"duration_ms":1234}

There are no named SSE event: fields. Everything is discriminated by the type key inside the JSON, so a client can parse each data: payload and switch on type.

Two chunk vocabularies

The chunk types depend on which family you called.

Meta-search: search/everywhere and search/news

Planning and fan-out telemetry interleaved with results: meta, source_started, source_pruned, items, source_failed, plan_refined, ranked_partial, ranked_final, comments_enriched, transcript_enriched, clusters, warning, and a terminal done (or error then done). The universal search reference documents each one with a real payload and the ordering guarantees.

Prism composites and batch: everything else in the table

A smaller vocabulary:

result.key is a per-endpoint label rather than an index: stat for post-stats rows, item for comment-lookup, profile for profiles, transcript for transcripts, and comments for prism/comments. A final legs frame carries the leg array on the endpoints that report one.

Terminal guarantees

• done is always last. If the handler forgets to emit one, the stream appends it before closing, so a client can always wait for done rather than for the socket to close.
• An error is a frame, not a dropped connection. A failure mid-run emits error and then done. The terminal summary reports committed billing state: a rejected immediate refund remains refunded:false with the retained charge while exact compensation is queued.
• Prism streams have a hard 55-second server-side ceiling. If the fan-out neither finishes nor settles within it, the stream attempts its refund, emits error then done, and closes rather than holding your connection open indefinitely. Each composite's own per-leg deadlines normally settle long before this fires, and batch endpoints additionally cut off at 45 seconds with deferred rows.

Billing on a stream

There is no envelope, so there is no credits_used field and no credits_remaining to read at the end of the body. The terminal done event carries the figures instead:

• Meta-search: summary.credits_used, plus refunded and partial_failure.
• Prism batch: the nine-field batch summary, including credits_charged and credits_refunded. It is the identical object the sync JSON response returns, so the two can never disagree.

Normal settlement and emergency recovery are serialized. If a timeout closes the stream while the executor is still unwinding, only one path can reduce the held charge; a late continuation exits instead of issuing a second refund.

Streams are never cached. A streaming request also cannot poison a later sync request's cache, because the two are keyed separately. Asking for a stream and then asking for JSON gives you a real second run rather than a replayed stream.

Concurrency and rate limits

A stream counts as one request against the 600 requests-per-minute budget, the same as any other call. Its concurrency slot is released once the response begins, not when the last frame lands, so a long-running stream does not hold one of your 50 concurrent slots open for its whole duration. See Rate limits.

When to stream

• Rendering results in a UI. search/everywhere takes roughly 12 to 30 seconds to complete but emits its first frame in about 300 ms. Streaming is the difference between a spinner and a list that fills in.
• Large batches. Processing rows as they settle beats blocking on the slowest item, and it means a deferred tail costs you nothing in wall-clock time.
• Long composites. prism/answers streams each engine's answer as it lands, which is why it has no sync form at all.

Stay on the sync JSON envelope for scheduled jobs, workflow tools, and anything that just wants the finished object. It is simpler to parse and gives you the credit fields directly.

Next steps