Tavily

Web grounding via the Tavily API. Search the current public web with optional LLM-generated answers, extract clean content from URLs, and crawl or map sitegraphs. Repeated searches may use the 2-minute endpoint cache; cache hits cost 0 credits, and Cache-Control: no-cache requests a billed refresh.

Base URL: /v1/tavily/...

Tavily uses the same authentication and credit model as every other SocialCrawl endpoint — pass your x-api-key header, pay 1 credit per call, get a unified response envelope. The fact that the upstream is POST + Bearer auth is invisible to you.

Getting Started

1. Run a web search

curl "https://www.socialcrawl.dev/v1/tavily/search?query=claude+opus+release+notes" \
  -H "x-api-key: sc_your_api_key_here"

2. Search with an LLM-generated answer

curl "https://www.socialcrawl.dev/v1/tavily/search?query=who+founded+anthropic&include_answer=true" \
  -H "x-api-key: sc_your_api_key_here"

The response includes both ranked results[] and a synthesised answer string.

3. Extract clean content from a URL

curl "https://www.socialcrawl.dev/v1/tavily/extract?urls=https://en.wikipedia.org/wiki/Lionel_Messi" \
  -H "x-api-key: sc_your_api_key_here"

You can pass up to 20 comma-separated URLs in a single call. Any that could not be read come back in failed_results rather than failing the request.

4. Walk a site

GET /v1/tavily/map returns the URLs a site exposes, without any page bodies — the cheap way to find out what exists before you spend anything reading it. GET /v1/tavily/crawl follows links from a starting URL and returns each page's extracted content.

# What pages does this site have?
curl "https://www.socialcrawl.dev/v1/tavily/map?url=https://docs.anthropic.com&max_depth=2&limit=100" \
  -H "x-api-key: sc_your_api_key_here"

# Read the ones about pricing — instructions steer which paths get followed
curl "https://www.socialcrawl.dev/v1/tavily/crawl?url=https://docs.anthropic.com&instructions=pages%20about%20pricing%20and%20rate%20limits&limit=20" \
  -H "x-api-key: sc_your_api_key_here"

Both share the same shaping parameters — max_depth, max_breadth, limit, select_paths, select_domains, exclude_paths, exclude_domains, allow_external, categories — so a map and the crawl that follows it can take identical scoping. What separates them is instructions: plain-language guidance on which paths are worth following, which is Tavily's distinguishing feature over a mechanical link walk. Use map when you want the shape of a site and crawl when you want its text.

Endpoints

4 endpoints available.

Endpoint	Path	Credit Tier
Crawl a website with LLM-driven path selection	`/v1/tavily/crawl`	standard (1cr)
Extract clean content from one or more URLs	`/v1/tavily/extract`	standard (1cr)
Map a website's sitegraph	`/v1/tavily/map`	standard (1cr)
Tavily web search with optional LLM-generated answer	`/v1/tavily/search`	standard (1cr)

Response Shape

Tavily endpoints use the Analytics archetype, which means the upstream response passes through with only the standard envelope-key strip applied. You see Tavily's full payload — including the answer field for search, the failed_results array for extract, and the instructions echo for crawl — exactly as Tavily returns it.

This differs from the social platforms (TikTok, Instagram, etc.) which apply a unified Author / Post field map. Web search results aren't social posts and aren't shoehorned into that schema.

Notes

All endpoints use GET method with query parameters
Authentication via x-api-key header (your SocialCrawl key, not your Tavily key)
Array-typed Tavily params (e.g. urls, include_domains, select_paths) are accepted as comma-separated strings on this surface and split server-side
Empty search results (results: []) are billed normally — you paid for the search, an empty result is a real answer
Caching: search responses cached 2 min, extract 10 min, map / crawl 30 min

Tavily

On this page