SocialCrawl
Cookbooks

Watch a topic and get only new posts, with what people say under them

Search several platforms for a topic, judge each post's stance, read the comments under the busiest ones, and repeat on a schedule that returns only new posts

Send your topic once and get back the last day's posts from several platforms. Each post carries its stance, and the busiest ones carry their comments. Then put the same request on a schedule, and every run returns only the posts the last one did not. The example watches what people say about oat milk in coffee.

Measured on 18/11/2026: one run kept 39 posts from the last day and returned 20, for 17 credits. That paid for 8 credits of searches, 2 of stance labels and 7 for 26 comments under 3 posts.

Start in five minutes

Set your key once. Every block on this page reads it.

Terminal
export SOCIALCRAWL_API_KEY=YOUR_API_KEY

Paste this prompt into your coding agent:

Prompt
Read https://www.socialcrawl.dev/docs/recipes/watch-a-topic.md and follow it to watch posts about oat milk and oat lattes on X, Reddit, Threads and news. Use the key in SOCIALCRAWL_API_KEY. Price the request with a dry run first (dry_run=true, which is free), and show me the estimate. Send max_credits=60 on every run, and ask me before you create a schedule.

Or price it yourself. A dry run plans the searches and quotes them for 0 credits:

cURL
curl -G "https://www.socialcrawl.dev/v1/prism/discover" \
  -H "x-api-key: $SOCIALCRAWL_API_KEY" \
  --data-urlencode "keywords=oat milk,oat latte" \
  --data-urlencode "platforms=twitter,reddit,threads,news" \
  --data-urlencode "period=1d" \
  --data-urlencode "label=stance" \
  --data-urlencode "comments=inline" \
  --data-urlencode "comments.top_posts=3" \
  --data-urlencode "limit=20" \
  --data-urlencode "max_credits=60" \
  --data-urlencode "dry_run=true"

On 18/11/2026 it quoted about 26 credits, at most 59.

How it works

  • Each keyword runs as one search on each platform you list.
  • Posts outside your window are dropped, and the same post found twice is kept once.
  • Each kept post is judged for its stance toward the topic.
  • The comments under the busiest posts are read into the post's row.
  • A schedule repeats the request and starts each run where the last one stopped.

What you need

CallWhat it givesCredits
GET /v1/prism/discoverOne run now, answered in the callEach search at its platform's price, plus labels and comments, up to max_credits
POST /v1/prism/discover with scheduleThe same request on a schedule0 to create, each run billed like the call above
GET /v1/events or a webhook endpointWord that a scheduled run has ended0

Steps

Step 1: run it once

cURL
curl -G "https://www.socialcrawl.dev/v1/prism/discover" \
  -H "x-api-key: $SOCIALCRAWL_API_KEY" \
  --data-urlencode "keywords=oat milk,oat latte" \
  --data-urlencode "platforms=twitter,reddit,threads,news" \
  --data-urlencode "period=1d" \
  --data-urlencode "label=stance" \
  --data-urlencode "comments=inline" \
  --data-urlencode "comments.top_posts=3" \
  --data-urlencode "limit=20" \
  --data-urlencode "max_credits=60"
Python
import json, os, urllib.parse, urllib.request

params = {"keywords": "oat milk,oat latte", "platforms": "twitter,reddit,threads,news", "period": "1d",
          "label": "stance", "comments": "inline", "comments.top_posts": "3", "limit": "20", "max_credits": "60"}
url = "https://www.socialcrawl.dev/v1/prism/discover?" + urllib.parse.urlencode(params)
req = urllib.request.Request(url, headers={"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]})
data = json.load(urllib.request.urlopen(req, timeout=120))["data"]
for post in data["items"]:
    stance = post["judgments"].get("stance") or {}
    comments = (post.get("comments") or {}).get("items") or []
    print(post["platform"], stance.get("label"), len(comments), post["url"])
print("save for next time:", data["next"]["watermark"])
  • period=1d keeps the posts of the last day. A watch that runs every six hours still sends 1d, and the watermark removes what it has already seen.
  • label=stance adds judgments.stance to each post, a label with its confidence. It costs 1 credit per started 25 posts.
  • comments=inline reads the comments under the busiest 3 posts into a comments block on each of them. News rows have no comments to read.
Response (trimmed)
{
  "data": {
    "items": [
      {
        "platform": "twitter",
        "url": "https://x.com/VeganRecovering/status/2123032624772120643",
        "published_at": "2026-11-18T12:36:36.000Z",
        "text": "The end of oat milk is, apparently, upon us.",
        "judgments": { "stance": { "label": "negative", "confidence": 0.73 } }
      }
    ],
    "summary": {
      "stance": { "positive": 17, "negative": 3, "no_opinion": 4, "uncertain": 15 }
    },
    "next": { "watermark": "eyJ2IjoxLCJ3Ijoi..." }
  }
}

summary.stance counts every kept post, not only the 20 returned. A post whose stance could not be told has a null label and counts as uncertain.

Step 2: put it on a schedule

Send the same fields as JSON with a schedule. The example keeps dry_run, so it prices the schedule and creates nothing. Remove dry_run to create it.

cURL
curl -X POST "https://www.socialcrawl.dev/v1/prism/discover" \
  -H "x-api-key: $SOCIALCRAWL_API_KEY" \
  -H "content-type: application/json" \
  -d '{"keywords":["oat milk","oat latte"],"platforms":["twitter","reddit","threads","news"],"period":"1d","label":["stance"],"comments":"inline","limit":20,"schedule":{"every":"6h","max_credits_per_run":60,"max_credits_per_month":3000},"dry_run":true}'
Python
import json, os, urllib.request

body = {"keywords": ["oat milk", "oat latte"], "platforms": ["twitter", "reddit", "threads", "news"],
        "period": "1d", "label": ["stance"], "comments": "inline", "limit": 20,
        "schedule": {"every": "6h", "max_credits_per_run": 60, "max_credits_per_month": 3000},
        "dry_run": True}  # remove to create the schedule
req = urllib.request.Request("https://www.socialcrawl.dev/v1/prism/discover", data=json.dumps(body).encode(),
                             headers={"x-api-key": os.environ["SOCIALCRAWL_API_KEY"], "content-type": "application/json"})
data = json.load(urllib.request.urlopen(req, timeout=120))["data"]
print(data["estimate"]["credits"], data["schedule"]["credits_per_month"], "credits a month at most")
  • every takes 1h, 6h, 12h, 1d, 7d or 30d.
  • max_credits_per_run caps each run, and max_credits_per_month pauses the schedule until the next month once a run could pass it.
  • The dry run answers what a month costs at most. Creating the schedule is free.

Step 3: hear when a run ends

Each scheduled run ends with an event. Read them from the event feed, or register a webhook endpoint for types such as discover.* and receive them as signed pushes. Each event names the run, and the run's rows are a free read.

Store only what is new

A schedule does this for you: each run starts from the previous run's watermark, so it returns only newer posts. When you run the request yourself instead, save data.next.watermark from each answer. Send it back as since_watermark next time. Key every post you store on its url, so a post that two keywords both found stays one row.

What it costs

PartCredits
The dry run0
Each search that returns postsThat platform's search price
Judging against your keywords0
Judging against a different context1 per started 15 posts
label=stance1 per started 25 posts
Comments under the busiest postsEach platform's comment page price
Creating a schedule0

What the example runs measured on 18/11/2026:

RunPostsCredits
The dry run00
The first run20 returned, 39 kept17
The schedule's dry run, every 6 hours00, and at most 7,080 a month
The full program a minute later, with the watermark0 new0

Tips, and what not to use

  • Leave context out until you need it. On this request the dry run priced it at 18 of about 37 credits, and judging against the keywords is free.
  • Set max_credits_per_month from the dry run's recommended_max_credits_per_month, so every run of the month can start.
  • Do not start a watch with GET /v1/search/everywhere. It costs 20 credits a search whatever it finds. In one measured watch, the same keywords cost 420 credits on it against 211 on Discover. Keep it for a one-off pulse check.

Full program and agent brief

Next steps