# Site links (/docs/recipes/links-to-a-site) Search X, Reddit and the web for posts that link to one domain, read each thread once, and keep every link the first time it appears Search each platform for your domain and get back the posts that carry a link to it. You keep every post once, and you know where each link first appeared. The example tracks `nasa.gov`. Measured on 18/11/2026: the four steps found 20 posts on X, 24 Reddit comments and 10 web pages for `nasa.gov`, and read one thread of 63 comments. That cost 8 credits in 4 paid calls. ## Start in five minutes Set your key once. Every block on this page reads it. ```bash title="Terminal" export SOCIALCRAWL_API_KEY=YOUR_API_KEY ``` Paste this prompt into your coding agent: ```text title="Prompt" Read https://www.socialcrawl.dev/docs/recipes/links-to-a-site.md and follow it to find the posts that link to nasa.gov. Use the key in SOCIALCRAWL_API_KEY. Price each paid call with a dry run first (add dry_run=1, which is free), and add up the prices. Keep the whole job under max_credits=20: stop and ask me before you go past it. ``` Or check the first call yourself. A dry run validates the request and prices it for 0 credits: ```bash title="cURL" curl -G "https://www.socialcrawl.dev/v1/twitter/search/tweets" \ -H "x-api-key: $SOCIALCRAWL_API_KEY" \ --data-urlencode 'query=url:"nasa.gov"' \ --data-urlencode "dry_run=1" ``` Remove `dry_run` to run it. It answers 20 posts for 1 credit. ## How it works - You search each platform for the address itself, such as `nasa.gov`. - X matches the address in a post's links, subdomains included. - Reddit and Google find comments and pages that spell the address out. - You read a whole thread only when a match in it is new to you. - You store each post once, keyed on its URL, with the link it carried. ## What you need | Call | What it gives | Credits | | --- | --- | --- | | `GET /v1/twitter/search/tweets` | Posts on X. `url:` keeps the ones that link to a domain | 1 a page | | `GET /v1/reddit/search/comments` | Reddit comments that contain the address | 1 a page | | `GET /v1/google/search` | Pages anywhere on the web that name the address | 1 a page | | `GET /v1/reddit/post/comments` | Every comment in one thread | 5 a page | ## Steps ### Step 1: posts on X that link to the site **cURL** ```bash title="cURL" curl -G "https://www.socialcrawl.dev/v1/twitter/search/tweets" \ -H "x-api-key: $SOCIALCRAWL_API_KEY" \ --data-urlencode 'query=url:"nasa.gov"' \ --data-urlencode "sort=latest" \ --data-urlencode "seen=nasa-links" ``` **Python** ```python title="Python" import json, os, urllib.parse, urllib.request params = {"query": 'url:"nasa.gov"', "sort": "latest", "seen": "nasa-links"} url = "https://www.socialcrawl.dev/v1/twitter/search/tweets?" + urllib.parse.urlencode(params) req = urllib.request.Request(url, headers={"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]}) data = json.load(urllib.request.urlopen(req, timeout=120))["data"] for row in data["items"]: print(row["post"]["published_at"], row["post"]["url"]) ``` - `url:"nasa.gov"` matches links to the domain and to its subdomains. In the run on 18/11/2026, all 20 posts carried a `nasa.gov` link, and one linked only to `www3.nasa.gov`. - `sort=latest` puts the newest posts first. Page on with `cursor`. - `seen` is a name you choose. Posts you already received under it are left out for 24 hours, and the page price falls with them. ```json title="Response (trimmed)" { "post": { "url": "https://x.com/nine8central_/status/2123034536548114924", "content": { "text": "Celebrating Voyager 1 reaching One Light Day from Earth. With https://NASA.gov light call sign animation https://t.co/vOHyioSi64" }, "published_at": "2026-11-18T12:44:12.000Z" } } ``` ### Step 2: Reddit comments that contain the address **cURL** ```bash title="cURL" curl -G "https://www.socialcrawl.dev/v1/reddit/search/comments" \ -H "x-api-key: $SOCIALCRAWL_API_KEY" \ --data-urlencode 'query="nasa.gov"' \ --data-urlencode "sort=new" ``` **Python** ```python title="Python" import json, os, urllib.parse, urllib.request params = {"query": '"nasa.gov"', "sort": "new"} url = "https://www.socialcrawl.dev/v1/reddit/search/comments?" + urllib.parse.urlencode(params) req = urllib.request.Request(url, headers={"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]}) data = json.load(urllib.request.urlopen(req, timeout=120))["data"] for row in data["items"]: c = row["comment"] print(c["published_at"], c["ext"]["subreddit"], c["url"]) ``` - Quote the address, so Reddit matches it as one phrase. - Each comment carries its thread's title in `ext.post_title` and its own `url`. The thread is the part of `url` before `/comment/`. ### Step 3: pages on the rest of the web **cURL** ```bash title="cURL" curl -G "https://www.socialcrawl.dev/v1/google/search" \ -H "x-api-key: $SOCIALCRAWL_API_KEY" \ --data-urlencode 'query="nasa.gov" -site:nasa.gov' \ --data-urlencode "date_posted=last-week" ``` **Python** ```python title="Python" import json, os, urllib.parse, urllib.request params = {"query": '"nasa.gov" -site:nasa.gov', "date_posted": "last-week"} url = "https://www.socialcrawl.dev/v1/google/search?" + urllib.parse.urlencode(params) req = urllib.request.Request(url, headers={"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]}) data = json.load(urllib.request.urlopen(req, timeout=120))["data"] for row in data["items"]: print(row["url"], row["title"]) ``` - `-site:nasa.gov` leaves out the site's own pages, so you see who else links to it. - `date_posted=last-week` keeps the pages Google dates in the last week. ### Step 4: read a thread once When a Reddit match is new, read its whole thread one time. Other comments in it can carry more links. **cURL** ```bash title="cURL" curl -G "https://www.socialcrawl.dev/v1/reddit/post/comments" \ -H "x-api-key: $SOCIALCRAWL_API_KEY" \ --data-urlencode "url=https://www.reddit.com/r/space/comments/1xzbpbk/" ``` **Python** ```python title="Python" import json, os, urllib.parse, urllib.request params = {"url": "https://www.reddit.com/r/space/comments/1xzbpbk/"} url = "https://www.socialcrawl.dev/v1/reddit/post/comments?" + urllib.parse.urlencode(params) req = urllib.request.Request(url, headers={"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]}) data = json.load(urllib.request.urlopen(req, timeout=120))["data"] print(len(data["items"]), "comments") ``` ## What you get back The full program at the foot of this page runs all four steps. It prints one line per new post, with the first link the post carried: ```text title="Terminal" reddit https://www.reddit.com/r/AskPhysics/comments/1xxxxae/comment/19ssqirrrl1hh/ -> https://science.nasa.gov/mission/parker-solar-probe/ web https://www.youtube.com/shorts/A-w4iprhpvw -> https://www.nasa.gov 46 new posts linking to nasa.gov; 5 credits ``` On 18/11/2026 its first run stored 46 posts. The three searches had just run, so they came from the cache for 0 credits, and the one thread it read cost 5. The next run prints only the posts it has not stored yet. ## Store only what is new Key each sighting on the post's URL. Before you store it, pull every link to the domain out of the text with a regular expression. Keep the earliest post for each link, and keep every later sighting too, so you can count how far a link spread. - On X, send the same `seen` name every run. Repeats drop out of the page, and a page of nothing but repeats is free. - On Reddit and Google, compare each URL with your table. - Mark each thread you read, so the next run never reads it again. ## What it costs | Step | Credits | | --- | --- | | A page of X posts | 1 | | A page of Reddit comments | 1 | | A page of Google results | 1 | | A Reddit thread read once | 5 a page | | A dry run of any of them | 0 | ## Tips, and what not to use - Reddit escapes underscores in comment text, so `iswa_data` arrives as `iswa\_data`. Remove the backslashes before you match. - A search that comes back empty is refunded. A page that holds only posts you already have is still billed, except on X with `seen`. - Do not start with `GET /v1/search/everywhere` for this job. It costs 20 credits a search and ranks posts by topic, whatever they link to. In one measured run for a single domain, 3 of its 40 results held the link. ## Full program and agent brief **Full program (Python)** ```python title="Python" # links_to_a_site.py: every new post that links to one site, stored once. # Run: SOCIALCRAWL_API_KEY=... python3 links_to_a_site.py (Python 3.9+, nothing to install) import json import os import re import sqlite3 import urllib.parse import urllib.request KEY = os.environ["SOCIALCRAWL_API_KEY"] BASE = "https://www.socialcrawl.dev/v1" SITE = "nasa.gov" THREADS_PER_RUN = 1 # each thread read costs 5 credits a page LINK = re.compile(r"https?://(?:[\w-]+\.)*" + re.escape(SITE) + r"[^\s)\]>\"']*", re.I) def get(path, **params): url = f"{BASE}/{path}?" + urllib.parse.urlencode(params) req = urllib.request.Request(url, headers={"x-api-key": KEY}) with urllib.request.urlopen(req, timeout=120) as res: body = json.load(res) return body["data"], body.get("credits_used", 0) db = sqlite3.connect("links.db") db.execute("CREATE TABLE IF NOT EXISTS sightings (post_url TEXT PRIMARY KEY, platform TEXT, published_at TEXT, links TEXT)") db.execute("CREATE TABLE IF NOT EXISTS threads (url TEXT PRIMARY KEY)") spent, new = 0, [] def keep(platform, post_url, published_at, text): links = sorted(set(LINK.findall((text or "").replace("\\_", "_")))) if not links: return False cur = db.execute("INSERT OR IGNORE INTO sightings VALUES (?, ?, ?, ?)", (post_url, platform, published_at, " ".join(links))) if cur.rowcount: new.append((platform, post_url, links[0])) return bool(cur.rowcount) data, c = get("twitter/search/tweets", query=f'url:"{SITE}"', sort="latest", seen=f"{SITE}-links") spent += c for row in data["items"]: p = row["post"] keep("x", p["url"], p["published_at"], p["content"]["text"]) data, c = get("reddit/search/comments", query=f'"{SITE}"', sort="new") spent += c fresh_threads = [] for row in data["items"]: cm = row["comment"] if keep("reddit", cm["url"], cm["published_at"], cm["text"]): fresh_threads.append(cm["url"].split("/comment/")[0] + "/") data, c = get("google/search", query=f'"{SITE}" -site:{SITE}', date_posted="last-week") spent += c for row in data["items"]: keep("web", row["url"], None, row["url"] + " " + (row.get("description") or "")) for thread in list(dict.fromkeys(fresh_threads)): if THREADS_PER_RUN <= 0: break if db.execute("SELECT 1 FROM threads WHERE url = ?", (thread,)).fetchone(): continue data, c = get("reddit/post/comments", url=thread) spent += c THREADS_PER_RUN -= 1 db.execute("INSERT INTO threads VALUES (?)", (thread,)) for row in data["items"]: cm = row["comment"] keep("reddit", cm["url"], cm["published_at"], cm["text"]) db.commit() for platform, url, link in new: print(f"{platform:7} {url} -> {link}") print(f"{len(new)} new posts linking to {SITE}; {spent} credits") ``` **Agent brief** ```text title="Agent brief" Goal: every new post that links to nasa.gov, stored once. Read https://www.socialcrawl.dev/docs/recipes/links-to-a-site.md first. 1. Dry-run each call with dry_run=1 and add up the prices. Stop if the total passes max_credits=20. 2. Search X with query url:"nasa.gov", sort=latest and seen=nasa-links. 3. Search Reddit comments for "nasa.gov" with sort=new, and Google for "nasa.gov" -site:nasa.gov with date_posted=last-week. 4. For each new Reddit match, read its thread once with reddit/post/comments. 5. Store each post once by URL, with every nasa.gov link in its text. Report only the new ones. ``` ## Next steps - [Watch a topic](/docs/recipes/watch-a-topic.md): New posts about a topic on a schedule, with what people say under them. - [X (Twitter)](/docs/twitter.md): Every X endpoint, its parameters and its price. - [Credits](/docs/credits.md): What a call costs and what is never charged.