SocialCrawl
Cookbooks

Find every post that links to a site

Search X, Reddit and the web for posts that link to one domain, read each thread once, and keep every link the first time it appears

Search each platform for your domain and get back the posts that carry a link to it. You keep every post once, and you know where each link first appeared. The example tracks nasa.gov.

Measured on 18/11/2026: the four steps found 20 posts on X, 24 Reddit comments and 10 web pages for nasa.gov, and read one thread of 63 comments. That cost 8 credits in 4 paid calls.

Start in five minutes

Set your key once. Every block on this page reads it.

Terminal
export SOCIALCRAWL_API_KEY=YOUR_API_KEY

Paste this prompt into your coding agent:

Prompt
Read https://www.socialcrawl.dev/docs/recipes/links-to-a-site.md and follow it to find the posts that link to nasa.gov. Use the key in SOCIALCRAWL_API_KEY. Price each paid call with a dry run first (add dry_run=1, which is free), and add up the prices. Keep the whole job under max_credits=20: stop and ask me before you go past it.

Or check the first call yourself. A dry run validates the request and prices it for 0 credits:

cURL
curl -G "https://www.socialcrawl.dev/v1/twitter/search/tweets" \
  -H "x-api-key: $SOCIALCRAWL_API_KEY" \
  --data-urlencode 'query=url:"nasa.gov"' \
  --data-urlencode "dry_run=1"

Remove dry_run to run it. It answers 20 posts for 1 credit.

How it works

  • You search each platform for the address itself, such as nasa.gov.
  • X matches the address in a post's links, subdomains included.
  • Reddit and Google find comments and pages that spell the address out.
  • You read a whole thread only when a match in it is new to you.
  • You store each post once, keyed on its URL, with the link it carried.

What you need

CallWhat it givesCredits
GET /v1/twitter/search/tweetsPosts on X. url: keeps the ones that link to a domain1 a page
GET /v1/reddit/search/commentsReddit comments that contain the address1 a page
GET /v1/google/searchPages anywhere on the web that name the address1 a page
GET /v1/reddit/post/commentsEvery comment in one thread5 a page

Steps

cURL
curl -G "https://www.socialcrawl.dev/v1/twitter/search/tweets" \
  -H "x-api-key: $SOCIALCRAWL_API_KEY" \
  --data-urlencode 'query=url:"nasa.gov"' \
  --data-urlencode "sort=latest" \
  --data-urlencode "seen=nasa-links"
Python
import json, os, urllib.parse, urllib.request

params = {"query": 'url:"nasa.gov"', "sort": "latest", "seen": "nasa-links"}
url = "https://www.socialcrawl.dev/v1/twitter/search/tweets?" + urllib.parse.urlencode(params)
req = urllib.request.Request(url, headers={"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]})
data = json.load(urllib.request.urlopen(req, timeout=120))["data"]
for row in data["items"]:
    print(row["post"]["published_at"], row["post"]["url"])
  • url:"nasa.gov" matches links to the domain and to its subdomains. In the run on 18/11/2026, all 20 posts carried a nasa.gov link, and one linked only to www3.nasa.gov.
  • sort=latest puts the newest posts first. Page on with cursor.
  • seen is a name you choose. Posts you already received under it are left out for 24 hours, and the page price falls with them.
Response (trimmed)
{
  "post": {
    "url": "https://x.com/nine8central_/status/2123034536548114924",
    "content": { "text": "Celebrating Voyager 1 reaching One Light Day from Earth. With https://NASA.gov light call sign animation https://t.co/vOHyioSi64" },
    "published_at": "2026-11-18T12:44:12.000Z"
  }
}

Step 2: Reddit comments that contain the address

cURL
curl -G "https://www.socialcrawl.dev/v1/reddit/search/comments" \
  -H "x-api-key: $SOCIALCRAWL_API_KEY" \
  --data-urlencode 'query="nasa.gov"' \
  --data-urlencode "sort=new"
Python
import json, os, urllib.parse, urllib.request

params = {"query": '"nasa.gov"', "sort": "new"}
url = "https://www.socialcrawl.dev/v1/reddit/search/comments?" + urllib.parse.urlencode(params)
req = urllib.request.Request(url, headers={"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]})
data = json.load(urllib.request.urlopen(req, timeout=120))["data"]
for row in data["items"]:
    c = row["comment"]
    print(c["published_at"], c["ext"]["subreddit"], c["url"])
  • Quote the address, so Reddit matches it as one phrase.
  • Each comment carries its thread's title in ext.post_title and its own url. The thread is the part of url before /comment/.

Step 3: pages on the rest of the web

cURL
curl -G "https://www.socialcrawl.dev/v1/google/search" \
  -H "x-api-key: $SOCIALCRAWL_API_KEY" \
  --data-urlencode 'query="nasa.gov" -site:nasa.gov' \
  --data-urlencode "date_posted=last-week"
Python
import json, os, urllib.parse, urllib.request

params = {"query": '"nasa.gov" -site:nasa.gov', "date_posted": "last-week"}
url = "https://www.socialcrawl.dev/v1/google/search?" + urllib.parse.urlencode(params)
req = urllib.request.Request(url, headers={"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]})
data = json.load(urllib.request.urlopen(req, timeout=120))["data"]
for row in data["items"]:
    print(row["url"], row["title"])
  • -site:nasa.gov leaves out the site's own pages, so you see who else links to it.
  • date_posted=last-week keeps the pages Google dates in the last week.

Step 4: read a thread once

When a Reddit match is new, read its whole thread one time. Other comments in it can carry more links.

cURL
curl -G "https://www.socialcrawl.dev/v1/reddit/post/comments" \
  -H "x-api-key: $SOCIALCRAWL_API_KEY" \
  --data-urlencode "url=https://www.reddit.com/r/space/comments/1xzbpbk/"
Python
import json, os, urllib.parse, urllib.request

params = {"url": "https://www.reddit.com/r/space/comments/1xzbpbk/"}
url = "https://www.socialcrawl.dev/v1/reddit/post/comments?" + urllib.parse.urlencode(params)
req = urllib.request.Request(url, headers={"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]})
data = json.load(urllib.request.urlopen(req, timeout=120))["data"]
print(len(data["items"]), "comments")

What you get back

The full program at the foot of this page runs all four steps. It prints one line per new post, with the first link the post carried:

Terminal
reddit  https://www.reddit.com/r/AskPhysics/comments/1xxxxae/comment/19ssqirrrl1hh/  ->  https://science.nasa.gov/mission/parker-solar-probe/
web     https://www.youtube.com/shorts/A-w4iprhpvw  ->  https://www.nasa.gov
46 new posts linking to nasa.gov; 5 credits

On 18/11/2026 its first run stored 46 posts. The three searches had just run, so they came from the cache for 0 credits, and the one thread it read cost 5. The next run prints only the posts it has not stored yet.

Store only what is new

Key each sighting on the post's URL. Before you store it, pull every link to the domain out of the text with a regular expression. Keep the earliest post for each link, and keep every later sighting too, so you can count how far a link spread.

  • On X, send the same seen name every run. Repeats drop out of the page, and a page of nothing but repeats is free.
  • On Reddit and Google, compare each URL with your table.
  • Mark each thread you read, so the next run never reads it again.

What it costs

StepCredits
A page of X posts1
A page of Reddit comments1
A page of Google results1
A Reddit thread read once5 a page
A dry run of any of them0

Tips, and what not to use

  • Reddit escapes underscores in comment text, so iswa_data arrives as iswa\_data. Remove the backslashes before you match.
  • A search that comes back empty is refunded. A page that holds only posts you already have is still billed, except on X with seen.
  • Do not start with GET /v1/search/everywhere for this job. It costs 20 credits a search and ranks posts by topic, whatever they link to. In one measured run for a single domain, 3 of its 40 results held the link.

Full program and agent brief

Next steps