Find every post that links to a site
Search X, Reddit and the web for posts that link to one domain, read each thread once, and keep every link the first time it appears
Search each platform for your domain and get back the posts that carry a link to it. You keep every post once, and you know where each link first appeared. The example tracks nasa.gov.
Measured on 18/11/2026: the four steps found 20 posts on X, 24 Reddit comments and 10 web pages for nasa.gov, and read one thread of 63 comments. That cost 8 credits in 4 paid calls.
Start in five minutes
Set your key once. Every block on this page reads it.
export SOCIALCRAWL_API_KEY=YOUR_API_KEYPaste this prompt into your coding agent:
Read https://www.socialcrawl.dev/docs/recipes/links-to-a-site.md and follow it to find the posts that link to nasa.gov. Use the key in SOCIALCRAWL_API_KEY. Price each paid call with a dry run first (add dry_run=1, which is free), and add up the prices. Keep the whole job under max_credits=20: stop and ask me before you go past it.Or check the first call yourself. A dry run validates the request and prices it for 0 credits:
curl -G "https://www.socialcrawl.dev/v1/twitter/search/tweets" \
-H "x-api-key: $SOCIALCRAWL_API_KEY" \
--data-urlencode 'query=url:"nasa.gov"' \
--data-urlencode "dry_run=1"Remove dry_run to run it. It answers 20 posts for 1 credit.
How it works
- You search each platform for the address itself, such as
nasa.gov. - X matches the address in a post's links, subdomains included.
- Reddit and Google find comments and pages that spell the address out.
- You read a whole thread only when a match in it is new to you.
- You store each post once, keyed on its URL, with the link it carried.
What you need
| Call | What it gives | Credits |
|---|---|---|
GET /v1/twitter/search/tweets | Posts on X. url: keeps the ones that link to a domain | 1 a page |
GET /v1/reddit/search/comments | Reddit comments that contain the address | 1 a page |
GET /v1/google/search | Pages anywhere on the web that name the address | 1 a page |
GET /v1/reddit/post/comments | Every comment in one thread | 5 a page |
Steps
Step 1: posts on X that link to the site
curl -G "https://www.socialcrawl.dev/v1/twitter/search/tweets" \
-H "x-api-key: $SOCIALCRAWL_API_KEY" \
--data-urlencode 'query=url:"nasa.gov"' \
--data-urlencode "sort=latest" \
--data-urlencode "seen=nasa-links"import json, os, urllib.parse, urllib.request
params = {"query": 'url:"nasa.gov"', "sort": "latest", "seen": "nasa-links"}
url = "https://www.socialcrawl.dev/v1/twitter/search/tweets?" + urllib.parse.urlencode(params)
req = urllib.request.Request(url, headers={"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]})
data = json.load(urllib.request.urlopen(req, timeout=120))["data"]
for row in data["items"]:
print(row["post"]["published_at"], row["post"]["url"])url:"nasa.gov"matches links to the domain and to its subdomains. In the run on 18/11/2026, all 20 posts carried anasa.govlink, and one linked only towww3.nasa.gov.sort=latestputs the newest posts first. Page on withcursor.seenis a name you choose. Posts you already received under it are left out for 24 hours, and the page price falls with them.
{
"post": {
"url": "https://x.com/nine8central_/status/2123034536548114924",
"content": { "text": "Celebrating Voyager 1 reaching One Light Day from Earth. With https://NASA.gov light call sign animation https://t.co/vOHyioSi64" },
"published_at": "2026-11-18T12:44:12.000Z"
}
}Step 2: Reddit comments that contain the address
curl -G "https://www.socialcrawl.dev/v1/reddit/search/comments" \
-H "x-api-key: $SOCIALCRAWL_API_KEY" \
--data-urlencode 'query="nasa.gov"' \
--data-urlencode "sort=new"import json, os, urllib.parse, urllib.request
params = {"query": '"nasa.gov"', "sort": "new"}
url = "https://www.socialcrawl.dev/v1/reddit/search/comments?" + urllib.parse.urlencode(params)
req = urllib.request.Request(url, headers={"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]})
data = json.load(urllib.request.urlopen(req, timeout=120))["data"]
for row in data["items"]:
c = row["comment"]
print(c["published_at"], c["ext"]["subreddit"], c["url"])- Quote the address, so Reddit matches it as one phrase.
- Each comment carries its thread's title in
ext.post_titleand its ownurl. The thread is the part ofurlbefore/comment/.
Step 3: pages on the rest of the web
curl -G "https://www.socialcrawl.dev/v1/google/search" \
-H "x-api-key: $SOCIALCRAWL_API_KEY" \
--data-urlencode 'query="nasa.gov" -site:nasa.gov' \
--data-urlencode "date_posted=last-week"import json, os, urllib.parse, urllib.request
params = {"query": '"nasa.gov" -site:nasa.gov', "date_posted": "last-week"}
url = "https://www.socialcrawl.dev/v1/google/search?" + urllib.parse.urlencode(params)
req = urllib.request.Request(url, headers={"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]})
data = json.load(urllib.request.urlopen(req, timeout=120))["data"]
for row in data["items"]:
print(row["url"], row["title"])-site:nasa.govleaves out the site's own pages, so you see who else links to it.date_posted=last-weekkeeps the pages Google dates in the last week.
Step 4: read a thread once
When a Reddit match is new, read its whole thread one time. Other comments in it can carry more links.
curl -G "https://www.socialcrawl.dev/v1/reddit/post/comments" \
-H "x-api-key: $SOCIALCRAWL_API_KEY" \
--data-urlencode "url=https://www.reddit.com/r/space/comments/1xzbpbk/"import json, os, urllib.parse, urllib.request
params = {"url": "https://www.reddit.com/r/space/comments/1xzbpbk/"}
url = "https://www.socialcrawl.dev/v1/reddit/post/comments?" + urllib.parse.urlencode(params)
req = urllib.request.Request(url, headers={"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]})
data = json.load(urllib.request.urlopen(req, timeout=120))["data"]
print(len(data["items"]), "comments")What you get back
The full program at the foot of this page runs all four steps. It prints one line per new post, with the first link the post carried:
reddit https://www.reddit.com/r/AskPhysics/comments/1xxxxae/comment/19ssqirrrl1hh/ -> https://science.nasa.gov/mission/parker-solar-probe/
web https://www.youtube.com/shorts/A-w4iprhpvw -> https://www.nasa.gov
46 new posts linking to nasa.gov; 5 creditsOn 18/11/2026 its first run stored 46 posts. The three searches had just run, so they came from the cache for 0 credits, and the one thread it read cost 5. The next run prints only the posts it has not stored yet.
Store only what is new
Key each sighting on the post's URL. Before you store it, pull every link to the domain out of the text with a regular expression. Keep the earliest post for each link, and keep every later sighting too, so you can count how far a link spread.
- On X, send the same
seenname every run. Repeats drop out of the page, and a page of nothing but repeats is free. - On Reddit and Google, compare each URL with your table.
- Mark each thread you read, so the next run never reads it again.
What it costs
| Step | Credits |
|---|---|
| A page of X posts | 1 |
| A page of Reddit comments | 1 |
| A page of Google results | 1 |
| A Reddit thread read once | 5 a page |
| A dry run of any of them | 0 |
Tips, and what not to use
- Reddit escapes underscores in comment text, so
iswa_dataarrives asiswa\_data. Remove the backslashes before you match. - A search that comes back empty is refunded. A page that holds only posts you already have is still billed, except on X with
seen. - Do not start with
GET /v1/search/everywherefor this job. It costs 20 credits a search and ranks posts by topic, whatever they link to. In one measured run for a single domain, 3 of its 40 results held the link.
