SocialCrawl
EnginesDiscover

How Discover decides which posts to keep

See how keywords match, how each post is scored against your question and how duplicates are removed

A Discover row is kept when it is inside your date window, passes the relevance check and is not a copy of a row already kept. This page explains each step, so you can tell why a post was kept or dropped. Dropped rows carry a drop_reason. Every other request field is on Discover request fields.

Is a keyword an exact match?

No, and not a wildcard either. Each keyword is sent to every platform's own search, which matches words the way that platform does: most return posts containing the words or close variants, ranked by the platform. With expand=1 to 3, discover also plans extra searches from your keywords and context. Then every row is read by the judge against your keywords, or against context when you send one, so rows about something else are dropped under relevance=filter. literal_match on a row says whether a keyword appears in its text as written, and exclude_terms and contains filter on the text exactly.

Score each post against your question

context says what kept rows must be about: a persona or a question goes there, never in the keywords. Each row is scored from 0 to 1 against it, in judgments.relevance.p. When context is the same as the keywords, the score is worked out by code and costs nothing. A context that differs is read by a model and billed, and a dry run shows that cost in by_stage.judge.

relevance decides what the score does. filter, the default, drops rows below relevance_threshold, which you can set anywhere from 0 to 1. score keeps every row with its score, so you can sort and cut yourself. off skips the check.

Add labels

label adds judgments to each row, and each one is billed: mention (it needs brands), intent, niche, sponsored_model or stance. intent also filters: intent=complaint,question keeps only rows with those intents. What each label means, and which labels add no credits, is on Labels.

Ask your own questions

questions takes up to five questions of your own, as JSON keyed by the answer's id. They are judged on every account, and on every row with view=posts. A question about a sensitive trait of a person is refused as a free 400.

Remove duplicates

De-duplication is done by code and is free. Rows with the same canonical URL are one row, however many searches or groups found them: the first one seen stays and gains the other copies' matched_keywords, and the other groups go into also_in_groups. A row whose text is nearly the same as a kept row is marked state: "duplicate" with drop_reason: "near_duplicate" and duplicate_of, so a story syndicated on five sites keeps one row. Two texts that differ only in their own numbers, such as the same notice for two dates, are not treated as copies. A row already shown to you is never taken back.

Next steps

On this page