Universal news search

Send one query to GET /v1/search/news and get back a planned, localized, multi-country news search in a single call. An LLM plans your query into search angles and localizes each angle into the language of every requested country edition, the lane fans out up to 12 parallel legs per engine, and the articles merge into one deduplicated list.

Two independent news indexes are available through engines: google (the default, 50 country editions) and bing (49 of them). They barely overlap. On measured fixtures across four locales, none of the articles the bing engine returned appeared anywhere in the google results for the same query, so asking for both roughly doubles the distinct articles and the distinct publishers a query returns. It is also how you get article text: the google engine rarely carries a snippet, the bing engine carries one on effectively every row.

Every leg reports truthful provenance: query_source tells you whether its keyword was translated, confirmed original, or a fallback_original after a localization failure. Every article carries engine and a published_at_precision of exact or approximate, because the two indexes do not report publish times with the same fidelity. Returns one JSON envelope by default, or a live SSE stream when you ask for one.

curl "https://www.socialcrawl.dev/v1/search/news?query=samsung+galaxy+launch&countries=KR,US,JP&engines=google,bing" \
  -H "x-api-key: YOUR_API_KEY"

This lane is metered, 2-62 credits, not flat. On the default engine a 2-credit base covers planning and localization, then 1 credit per leg that returned at least one article, with an upfront hold of 2 + min(5 x countries, max_legs, 12) and a ceiling of 14. Adding the bing engine adds 1 credit per 5 articles that engine returns, because that index is priced per article rather than per call. The unused remainder of the hold is refunded automatically, so the credits_used you see in the response is what you actually paid.

How a request becomes legs

1. Plan. One combined LLM call turns your query into up to 5 weighted search angles and localizes each angle into the default language of every requested country. The call is fail-open: on timeout or error the lane degrades to a single identity angle (your query, clipped to a keyword) and every leg is stamped fallback_original, never silently.
2. Expand. Angles cross with countries into legs, capped at min(5 x countries, max_legs, 12). Identical (country, language, keyword) legs collapse before dispatch, so two angles that localize to the same keyword never bill twice.
3. Fan out. Every leg runs in parallel, once per engine you asked for. A google leg is an in-process Google News call using the primary source with automatic fallback and empty-result rescue, unless the request is pinned. A bing leg is a single call to the second index, paced at 6 concurrent legs.
4. Assemble. Articles dedupe across legs and across engines on canonical_url ?? url. The first leg's copy wins the body, and every later leg that returned the same article is appended to that item's legs_returned provenance list. sort orders the merged list.

The query_source vocabulary

Every leg carries one of three values, and they are part of the public contract:

Because original is never stamped on a failure path, a total localizer outage is always distinguishable from "keyword already correct".

Request parameters

publisher, from, and to pin every google leg to the primary Google News source: the fallback source cannot express those filters, and silently dropping a filter would return unfiltered data. Pinned requests trade away the fallback's empty-result rescue for filter correctness.

Those three filters are google-only. The bing index cannot express them, so a request that asks for them without google in engines is rejected with a 400 rather than quietly returning unfiltered results under a filter you believe was applied.

What you get back

By default (Accept: application/json) you get one envelope. data carries:

Two outcomes worth handling explicitly:

• Zero articles with succeeded legs is a 200 success, with an empty items and base-only billing. It is not an error.
• All legs failed is a 502 ALL_LEGS_FAILED with ok: false, and the full hold is refunded automatically. The 502 body is the standard error envelope on its own: per-leg detail (legs, plan) is not returned on this path, so ask for the SSE stream when you need to see which legs failed and why.

Streaming

Add Accept: text/event-stream and you get a live stream instead of one envelope.

{ "type": "meta",           "query": "...", "plan": {...}, "sources_planned": ["KR", "US", ...] }
{ "type": "plan_refined",   "plan": {...}, "legs": [{ "leg_id": "sq0|KR", "country": "KR", "language": "ko", "keyword": "...", "query_source": "translated", "angle_id": "sq0", "angle_label": "..." }] }
{ "type": "source_started", "source": "sq0|KR" }
{ "type": "items",          "source": "sq0|KR", "items": [...], "duration_ms": 1234 }
{ "type": "source_failed",  "source": "sq1|US", "error": { "code": "...", "message": "..." } }
{ "type": "done",           "summary": { "credits_used": 4, "refunded": true, "coverage": 1, ... } }

Order and guarantees:

• meta is first, carrying the deterministic identity plan before the LLM call resolves.
• plan_refined is always emitted on this lane, unlike universal social search, where it is conditional on the LLM changing the plan. It carries the final plan plus the expanded legs[] with per-leg query_source, so a consumer can render localization honesty (translated versus fallback chips) at plan time, before any leg settles. A degraded run still emits it, with every leg stamped fallback_original.
• Legs are keyed as source strings in the form angleId|COUNTRY, for example sq0|KR. Angle ids are positional with an sq prefix. These are per-request keys, not the fixed source ids of universal social search, so that endpoint's client reducer is not suitable for news streams. Skipped legs, aborted before dispatch, surface as source_failed with error.code: "aborted".
• items yield in completion order, so fast editions land first. The deduplicated cross-leg view is only available in sync mode, or by deduplicating client-side on canonical_url ?? url.
• done is always last, with credits_used (net), refunded, coverage, sources_called / sources_succeeded / sources_failed, total_items, and partial_failure.

On a stream, done is the only place the net charge appears. Response headers are flushed before the run settles, so a streaming response carries no X-Credits-Used or X-Credits-Remaining header. Sync mode does carry them.

Streaming responses are never cached and never touch the sync cache. They are also not replayable with an Idempotency-Key: replay applies to sync JSON responses only, and an SSE-accept request that reuses a spent key receives the stored JSON body back instead of a stream.

Billing

Metered, 2-62 credits. A 2-credit base covers planning and localization. Each google leg that returned at least one article is 1 credit; each bing leg is 1 credit per 5 articles it returned, because that index is priced per article rather than per call. The upfront hold is the base plus each planned leg's per-engine cost, which is 2 + min(5 x countries, max_legs, 12) on the default engine, and the unused remainder is refunded automatically.

X-Credits-Used on the response, and done.summary.credits_used on a stream, is the source of truth for what you paid. Use max_legs to cap the worst case: max_legs=3 caps the hold at 5 credits regardless of how many countries you request.

Billing on a stream is charge-for-delivered. If you disconnect mid-stream you pay the base plus the productive legs whose items chunks you received, and the rest of the hold is refunded. On an internal error the whole hold is refunded and the terminal done reports credits_used: 0, refunded: true.

Latency and failure modes

The lane plans before it fans out, so the combined plan-and-localize call is on the critical path. Its budget scales with the number of distinct languages (8s base plus 0.7s per extra language, capped at 18s). On timeout the run degrades to the identity plan with every leg stamped fallback_original and a note in plan.notes. It does not fail. Legs then run fully in parallel, each bounded by the remaining route budget. Typical end to end: a few seconds for one country, 10 to 30 seconds for a wide multi-language fan-out.

• One leg times out or errors: its row reports status: "failed" with an error class, sibling legs continue, and the leg bills 0.
• The localizer drops a language (operator-laden, overlong, or script-mismatched keyword): that leg searches the planned keyword stamped fallback_original.
• Client disconnect over SSE: work stops, undelivered legs are refunded, and the charge is settled exactly once.

Supported countries are the 50 editions in the lane's catalog, and an unknown countries code returns an error message carrying the up-to-date list. Notable exclusions: Russia and Lebanon are not servable by the primary news source and are not in the catalog. The bing engine serves 49 of the 50; CN is skipped on that engine, for free.

Adding the bing engine adds latency: measured end to end, a two-country search went from roughly 7 seconds on the default engine to between 8 and 24 seconds with both, because the second index is slower and is paced at 6 concurrent legs. Both stay inside the route budget. If a bing leg is throttled or fails, the google legs still answer, the call is a 200 partial, and the failed leg bills nothing.

Next steps