Tavily
Web grounding via the Tavily API. Search, extract, map, and crawl
Search the current public web with optional LLM-generated answers, extract clean content from URLs, and crawl or map sitegraphs. Every endpoint is a GET and costs 1 credit.
Base URL: /v1/tavily/...
Tavily endpoints use the Analytics archetype: the upstream response passes
through with only the standard envelope-key strip applied. You see Tavily's
full payload rather than the unified Author / Post field map that the
social platforms use. Web search results are not social posts.
Quickstart
1. Run a web search
curl "https://www.socialcrawl.dev/v1/tavily/search?query=claude+opus+release+notes" \
-H "x-api-key: YOUR_API_KEY"2. Ask for a written answer alongside the results
curl "https://www.socialcrawl.dev/v1/tavily/search?query=who+founded+anthropic&include_answer=true" \
-H "x-api-key: YOUR_API_KEY"The response includes both ranked results[] and a synthesised answer string.
Search and extract
| Endpoint | Credits | What it returns | Key parameters |
|---|---|---|---|
GET /v1/tavily/search | 1 | Ranked web results, plus an optional LLM-generated answer string | query, search_depth, topic, time_range, max_results, include_answer, include_domains, exclude_domains, country, start_date, end_date |
GET /v1/tavily/extract | 1 | Clean extracted content for one or more URLs, with unreadable ones in failed_results | urls (up to 20, comma-separated), extract_depth, format, include_images, timeout |
Any URL that could not be read comes back in failed_results rather than failing the whole request.
curl "https://www.socialcrawl.dev/v1/tavily/extract?urls=https://en.wikipedia.org/wiki/Lionel_Messi" \
-H "x-api-key: YOUR_API_KEY"Walk a site
| Endpoint | Credits | What it returns | Key parameters |
|---|---|---|---|
GET /v1/tavily/map | 1 | The URLs a site exposes, without any page bodies | url, max_depth, max_breadth, limit, instructions, select_paths, select_domains, exclude_paths, exclude_domains, allow_external, categories |
GET /v1/tavily/crawl | 1 | Each followed page's extracted content, starting from one URL | The same shaping parameters, plus extract_depth and format |
Both share the same scoping parameters, so a map and the crawl that follows it can take identical settings. What separates them is instructions: plain-language guidance on which paths are worth following, which is Tavily's distinguishing feature over a mechanical link walk.
Use map when you want the shape of a site, and crawl when you want its text. Mapping first is the cheap way to find out what exists before you spend anything reading it.
# What pages does this site have?
curl "https://www.socialcrawl.dev/v1/tavily/map?url=https://docs.anthropic.com&max_depth=2&limit=100" \
-H "x-api-key: YOUR_API_KEY"
# Read the ones about pricing
curl "https://www.socialcrawl.dev/v1/tavily/crawl?url=https://docs.anthropic.com&instructions=pages%20about%20pricing%20and%20rate%20limits&limit=20" \
-H "x-api-key: YOUR_API_KEY"All endpoints
4 endpoints available.
| Endpoint | Path | Credit Tier |
|---|---|---|
| Crawl a website with LLM-driven path selection | /v1/tavily/crawl | standard (1cr) |
| Extract clean content from one or more URLs | /v1/tavily/extract | standard (1cr) |
| Map a website's sitegraph | /v1/tavily/map | standard (1cr) |
| Tavily web search with optional LLM-generated answer | /v1/tavily/search | standard (1cr) |
Platform notes
- Empty search results (
results: []) are billed normally. You paid for the search, and an empty result is a real answer. - Array-typed Tavily params (
urls,include_domains,select_paths, and the rest) are accepted as comma-separated strings on this surface and split server-side. - Authentication is your SocialCrawl
x-api-key, not a Tavily key. - Caching: search responses cache for 2 minutes, extract for 10 minutes, map and crawl for 30 minutes. Cache hits cost 0 credits, and
Cache-Control: no-cacherequests a billed refresh.