Market research

A client research job asks the same questions about several topics in several markets. Discover runs the whole grid as one background run: each topic and market is a group with its own keywords, sources and tags, every row says which group found it, and the run keeps a log of every row it dropped and every call it made.

The request

{
  "groups": [
    {
      "id": "hyperhidrosis-KR",
      "keywords": ["다한증", "손 다한증", "다한증 수술 후기"],
      "context": "patients and carers asking about or describing hyperhidrosis treatment",
      "preset": "korea",
      "languages": ["ko"],
      "country": "KR",
      "sites": ["dcinside.com", "fmkorea.com", "theqoo.net"],
      "tags": { "topic": "hyperhidrosis", "market": "KR" }
    },
    {
      "id": "hyperhidrosis-JP",
      "keywords": ["多汗症", "手汗 治療"],
      "platforms": ["youtube", "twitter", "web"],
      "sites": ["detail.chiebukuro.yahoo.co.jp"],
      "languages": ["ja"],
      "country": "JP",
      "tags": { "topic": "hyperhidrosis", "market": "JP" }
    }
  ],
  "period": "90d",
  "outside_window": "keep",
  "min_text_length": 12,
  "pages_per_platform": 10,
  "include": ["dropped", "body"],
  "body_limit": 2000,
  "max_credits": 8000
}

# request.json is the body above: "groups": [...], "max_credits": 8000
curl -X POST "https://www.socialcrawl.dev/v1/prism/discover" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  --data @request.json

Send it with "dry_run": true first: the answer is the plan of every search and the estimate of credits and seconds, free. The real request answers 202 with the run. Follow data.next.status_url (it waits up to 25 seconds) until the status is completed, then download every row:

curl "https://www.socialcrawl.dev/v1/prism/discover/runs/{run_id}/results.jsonl?include=dropped" \
  -H "x-api-key: YOUR_API_KEY" > rows.jsonl

What each part does

Groups and tags. Each group is one topic in one market, with its own keywords, context, sources (preset or platforms), languages, country and sites. Up to 10 tags are echoed on every row of the group as group_tags, so a dashboard pivots on topic and market without parsing ids. A row found by two groups carries the first group's tags and lists the others in also_in_groups. A search shared by groups is paid once. Read one cell of the grid with GET /v1/prism/discover/runs/{run_id}?tag.market=JP, and data.run.credits.by_group says what each group cost.

The korea preset. It searches Naver (blogs, cafes, KnowledgeiN and news, one search per corpus), YouTube, TikTok, Instagram, Threads, news and the web. With a window, each dated Naver corpus reads its best 100 rows by relevance plus one newest-first page; search_in.naver=blog,cafe,kin,news,webkr adds Naver's web index to the four. Search in Korean: an English keyword finds little on Naver.

sites. Each named domain becomes site-scoped web searches from the group's own market, merged into the web source. include=body reads the full text of the top body_limit kept pages, and body_status says what each read gave: full, login_wall, paywall_preview, consent_wall, challenge, empty, error or list_page. A page behind a login wall, a block, a challenge or an empty page costs nothing and keeps its search snippet with text_truncated: true; a paywall preview, a consent wall or a list page is billed as served. data.sites reports, for every domain the run touched, the rows kept, the bodies by status, the dated share and whether a proxy was needed.

Estimated dates. KnowledgeiN publishes no date, so the Naver lane estimates one from the document id (the published_at_est field on /v1/naver/kin/search). In discover the estimate becomes published_at with date_source: "estimate", date_basis: "kin_docid_estimate" and in_window_estimated: true, and summary counts these rows apart from real dates. estimated_dates=undated treats them as undated and estimated_dates=drop drops them. Naver cafe rows carry no date at all and are counted apart with in_window: null.

Rows outside the window. outside_window=keep keeps rows dated before or after the window, marked in_window: false and counted apart, so a research run can still read older material while it measures what is recent. It buys no extra pages. min_text_length=12 drops near-empty rows before they are judged, so they are never billed for judging.

The audit trail

Start the run with include=dropped and every dropped row keeps its full evidence, with state and drop_reason: off_topic (with judgments.relevance.p), out_of_window, estimated_date, excluded_term, filter:<name>, empty (under min_text_length), duplicate or near_duplicate (both with duplicate_of). Without it, a dropped row is only { id, platform, state, drop_reason, duplicate_of }, and its text is deleted after the run ends. Read them with GET /v1/prism/discover/runs/{run_id}?state=dropped (or state=all), or with include=dropped on the run route or results.jsonl, which returns kept, dropped and duplicate rows together in arrival order. The GET answer itself carries kept rows only. Reading is free, and include=dropped on the run costs nothing.

Malformed rows and exact duplicates are counted in summary.rows (malformed, exact_duplicates) rather than listed, and suppressed rows are never listed. The books balance: kept, dropped and duplicate rows plus summary.rows.malformed plus summary.rows.exact_duplicates equal summary.rows.returned.

GET /v1/prism/discover/runs/{run_id}/legs lists every search the run made, free and paged: platform, lane, endpoint, search text, page, status, request_id, credits and error code. Every row's provenance is { endpoint, request_id, captured_at }, and its request_id is the request_id of one leg, so each row traces to the one call that returned it, and each leg's credits are what that call billed.

Site access

Measured on 16/10/2026 with sites and include=body on one hyperhidrosis query per market. The samples are small (the "Bodies read" column), so read the table as what to expect, and read each run's own data.sites, which reports the same columns for every domain it touched.

"Dated share" is the share of kept rows from that domain that carry a date, the page's own date included, so on a forum with a low share most rows are counted apart as undated. A list page is a forum or category index rather than one post, so the row keeps its search snippet. A known board-list address (a gallery list, a category or search page) is skipped before any read and costs nothing; a page that turns out to be a list after it is read is billed for that one read. Paywall previews and consent walls are billed as served, while login walls, blocks, challenges and empty pages cost nothing. Blind (teamblind.com) served full text for indexed posts, so it is not a login wall.

What discover cannot read

Community content behind a member login is never read (limit L11): Naver cafe full text and dates, and any site the table above marks as a wall. Those rows keep the search snippet, marked text_truncated: true with body_status: login_wall, and cost nothing to try. KnowledgeiN and blogs often carry the same questions with full text.

What a research job still calls directly

Discover searches what people publish about a topic. A research job usually adds a few direct calls beside the run:

The korea-community-research and korea-site-research recipes (GET /v1/utility/recipes) call the same lanes one at a time, which suits a single topic on a single source.

Next steps