# Market research (/docs/prism/discover-research)

Run a topic and market grid as one discover run, with Korean sources, named sites, estimated dates and an audit trail of every drop

A client research job asks the same questions about several topics in several markets. Discover runs the whole grid as one background run: each topic and market is a group with its own keywords, sources and tags, every row says which group found it, and the run keeps a log of every row it dropped and every call it made.

## The request

```json title="Request body"
{
  "groups": [
    {
      "id": "hyperhidrosis-KR",
      "keywords": ["다한증", "손 다한증", "다한증 수술 후기"],
      "context": "patients and carers asking about or describing hyperhidrosis treatment",
      "preset": "korea",
      "languages": ["ko"],
      "country": "KR",
      "sites": ["dcinside.com", "fmkorea.com", "theqoo.net"],
      "tags": { "topic": "hyperhidrosis", "market": "KR" }
    },
    {
      "id": "hyperhidrosis-JP",
      "keywords": ["多汗症", "手汗 治療"],
      "platforms": ["youtube", "twitter", "web"],
      "sites": ["detail.chiebukuro.yahoo.co.jp"],
      "languages": ["ja"],
      "country": "JP",
      "tags": { "topic": "hyperhidrosis", "market": "JP" }
    }
  ],
  "period": "90d",
  "outside_window": "keep",
  "min_text_length": 12,
  "pages_per_platform": 10,
  "include": ["dropped", "body"],
  "body_limit": 2000,
  "max_credits": 8000
}
```

```bash title="cURL"
# request.json is the body above: "groups": [...], "max_credits": 8000
curl -X POST "https://www.socialcrawl.dev/v1/prism/discover" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  --data @request.json
```

Send it with `"dry_run": true` first: the answer is the plan of every search and the estimate of credits and seconds, free. The real request answers `202` with the run. Follow `data.next.status_url` (it waits up to 25 seconds) until the status is `completed`, then download every row:

```bash title="cURL"
curl "https://www.socialcrawl.dev/v1/prism/discover/runs/{run_id}/results.jsonl?include=dropped" \
  -H "x-api-key: YOUR_API_KEY" > rows.jsonl
```

## What each part does

**Groups and tags.** Each group is one topic in one market, with its own `keywords`, `context`, sources (`preset` or `platforms`), `languages`, `country` and `sites`. Up to 10 `tags` are echoed on every row of the group as `group_tags`, so a dashboard pivots on topic and market without parsing ids. A row found by two groups carries the first group's tags and lists the others in `also_in_groups`. A search shared by groups is paid once. Read one cell of the grid with `GET /v1/prism/discover/runs/{run_id}?tag.market=JP`, and `data.run.credits.by_group` says what each group cost.

**The `korea` preset.** It searches Naver (blogs, cafes, KnowledgeiN and news, one search per corpus), YouTube, TikTok, Instagram, Threads, news and the web. With a window, each dated Naver corpus reads its best 100 rows by relevance plus one newest-first page; `search_in.naver=blog,cafe,kin,news,webkr` adds Naver's web index to the four. Search in Korean: an English keyword finds little on Naver.

**`sites`.** Each named domain becomes site-scoped web searches from the group's own market, merged into the `web` source. `include=body` reads the full text of the top `body_limit` kept pages, and `body_status` says what each read gave: `full`, `login_wall`, `paywall_preview`, `consent_wall`, `challenge`, `empty`, `error` or `list_page`. A page behind a login wall, a block, a challenge or an empty page costs nothing and keeps its search snippet with `text_truncated: true`; a paywall preview, a consent wall or a list page is billed as served. `data.sites` reports, for every domain the run touched, the rows kept, the bodies by status, the dated share and whether a proxy was needed.

**Estimated dates.** KnowledgeiN publishes no date, so the Naver lane estimates one from the document id (the `published_at_est` field on `/v1/naver/kin/search`). In discover the estimate becomes `published_at` with `date_source: "estimate"`, `date_basis: "kin_docid_estimate"` and `in_window_estimated: true`, and `summary` counts these rows apart from real dates. `estimated_dates=undated` treats them as undated and `estimated_dates=drop` drops them. Naver cafe rows carry no date at all and are counted apart with `in_window: null`.

**Rows outside the window.** `outside_window=keep` keeps rows dated before or after the window, marked `in_window: false` and counted apart, so a research run can still read older material while it measures what is recent. It buys no extra pages. `min_text_length=12` drops near-empty rows before they are judged, so they are never billed for judging.

## The audit trail

Start the run with `include=dropped` and every dropped row keeps its full evidence, with `state` and `drop_reason`: `off_topic` (with `judgments.relevance.p`), `out_of_window`, `estimated_date`, `excluded_term`, `filter:<name>`, `empty` (under `min_text_length`), `duplicate` or `near_duplicate` (both with `duplicate_of`). Without it, a dropped row is only `{ id, platform, state, drop_reason, duplicate_of }`, and its text is deleted after the run ends. Read them with `GET /v1/prism/discover/runs/{run_id}?state=dropped` (or `state=all`), or with `include=dropped` on the run route or `results.jsonl`, which returns kept, dropped and duplicate rows together in arrival order. The GET answer itself carries kept rows only. Reading is free, and `include=dropped` on the run costs nothing.

Malformed rows and exact duplicates are counted in `summary.rows` (`malformed`, `exact_duplicates`) rather than listed, and suppressed rows are never listed. The books balance: kept, dropped and duplicate rows plus `summary.rows.malformed` plus `summary.rows.exact_duplicates` equal `summary.rows.returned`.

`GET /v1/prism/discover/runs/{run_id}/legs` lists every search the run made, free and paged: platform, lane, endpoint, search text, page, status, `request_id`, credits and error code. Every row's `provenance` is `{ endpoint, request_id, captured_at }`, and its `request_id` is the `request_id` of one leg, so each row traces to the one call that returned it, and each leg's credits are what that call billed.

## Site access

Measured on 16/10/2026 with `sites` and `include=body` on one hyperhidrosis query per market. The samples are small (the "Bodies read" column), so read the table as what to expect, and read each run's own `data.sites`, which reports the same columns for every domain it touched.

| Domain | Market | Bodies read | What the reads gave | Dated share | Proxy needed |
| --- | --- | --- | --- | --- | --- |
| `gall.dcinside.com` | KR | 2 | Full text 2 of 2 | 14% | No |
| `fmkorea.com` | KR | 4 | No full text: 2 list pages, 2 errors | 44% | Yes, and it still failed |
| `theqoo.net` | KR | 2 | Full text 1, list page 1 | 9% | No |
| `82cook.com` | KR | 3 | Full text 3 of 3 | 33% | No |
| `teamblind.com` | KR | 2 | Full text 2 of 2 | 20% | No |
| `detail.chiebukuro.yahoo.co.jp` | JP | 4 | Full text 4 of 4 | 50% | No |
| `medium.com` | US | 3 | Full text 3 of 3 | 100% | No |
| `*.substack.com` | US | 4 | Full text 3, paywall preview 1 | 88% | No |
| `patient.info` and `community.patient.info` | US | 3 | Full text 3 of 3 | 67% | No |
| Plain web, 10 health sites | US | 4 | Full text 4 of 4 | 10% | No |
| Plain web, 10 health sites | GB | 4 | Full text 2, list page 1, error 1 | 30% | No |

"Dated share" is the share of kept rows from that domain that carry a date, the page's own date included, so on a forum with a low share most rows are counted apart as undated. A list page is a forum or category index rather than one post, so the row keeps its search snippet. A known board-list address (a gallery list, a category or search page) is skipped before any read and costs nothing; a page that turns out to be a list after it is read is billed for that one read. Paywall previews and consent walls are billed as served, while login walls, blocks, challenges and empty pages cost nothing. Blind (`teamblind.com`) served full text for indexed posts, so it is not a login wall.

## What discover cannot read

Community content behind a member login is never read (limit L11): Naver cafe full text and dates, and any site the table above marks as a wall. Those rows keep the search snippet, marked `text_truncated: true` with `body_status: login_wall`, and cost nothing to try. KnowledgeiN and blogs often carry the same questions with full text.

## What a research job still calls directly

Discover searches what people publish about a topic. A research job usually adds a few direct calls beside the run:

| Need | Call |
| --- | --- |
| The client's own site and channels | Each platform's profile and posts endpoints, such as `/v1/instagram/profile/posts`, and `/v1/web/crawl` for the site. |
| Price tables from a page | `/v1/web/extract` with the fields you want. |
| A brand's ads by page | `/v1/facebook/adlibrary/company/ads`. |
| How AI assistants describe the brand | `/v1/prism/ai-visibility`. |
| Questions people ask about a topic | `/v1/prism/audience-questions`. |
| Search interest over time in Korea | `/v1/naver/search-trend`. |
| Your own labels on every row (voice, role, treatment questions) | Run them in your own pipeline on `results.jsonl`. |

The `korea-community-research` and `korea-site-research` recipes (`GET /v1/utility/recipes`) call the same lanes one at a time, which suits a single topic on a single source.

## Next steps

- [Discover](/docs/prism/discover.md)} description="Every field, status, price and limit of discover." />
- [Discover for agents](/docs/prism/discover-agents.md)} description="Tools, retries and long runs for AI agents." />