# Tavily (/docs/tavily)



Tavily [#tavily]

Web grounding via the Tavily API. Search the current public web with optional LLM-generated answers, extract clean content from URLs, and crawl or map sitegraphs. Repeated searches may use the 2-minute endpoint cache; cache hits cost 0 credits, and `Cache-Control: no-cache` requests a billed refresh.

Base URL: `/v1/tavily/...`

Tavily uses the same authentication and credit model as every other SocialCrawl endpoint — pass your `x-api-key` header, pay 1 credit per call, get a unified response envelope. The fact that the upstream is POST + Bearer auth is invisible to you.

Getting Started [#getting-started]

1\. Run a web search [#1-run-a-web-search]

```bash
curl "https://www.socialcrawl.dev/v1/tavily/search?query=claude+opus+release+notes" \
  -H "x-api-key: sc_your_api_key_here"
```

2\. Search with an LLM-generated answer [#2-search-with-an-llm-generated-answer]

```bash
curl "https://www.socialcrawl.dev/v1/tavily/search?query=who+founded+anthropic&include_answer=true" \
  -H "x-api-key: sc_your_api_key_here"
```

The response includes both ranked `results[]` and a synthesised `answer` string.

3\. Extract clean content from a URL [#3-extract-clean-content-from-a-url]

```bash
curl "https://www.socialcrawl.dev/v1/tavily/extract?urls=https://en.wikipedia.org/wiki/Lionel_Messi" \
  -H "x-api-key: sc_your_api_key_here"
```

You can pass up to 20 comma-separated URLs in a single call. Any that could not be read come back in `failed_results` rather than failing the request.

4\. Walk a site [#4-walk-a-site]

`GET /v1/tavily/map` returns the URLs a site exposes, without any page bodies — the cheap way to find out what exists before you spend anything reading it. `GET /v1/tavily/crawl` follows links from a starting URL and returns each page's extracted content.

```bash
# What pages does this site have?
curl "https://www.socialcrawl.dev/v1/tavily/map?url=https://docs.anthropic.com&max_depth=2&limit=100" \
  -H "x-api-key: sc_your_api_key_here"

# Read the ones about pricing — instructions steer which paths get followed
curl "https://www.socialcrawl.dev/v1/tavily/crawl?url=https://docs.anthropic.com&instructions=pages%20about%20pricing%20and%20rate%20limits&limit=20" \
  -H "x-api-key: sc_your_api_key_here"
```

Both share the same shaping parameters — `max_depth`, `max_breadth`, `limit`, `select_paths`, `select_domains`, `exclude_paths`, `exclude_domains`, `allow_external`, `categories` — so a map and the crawl that follows it can take identical scoping. What separates them is `instructions`: plain-language guidance on which paths are worth following, which is Tavily's distinguishing feature over a mechanical link walk. Use `map` when you want the shape of a site and `crawl` when you want its text.

Endpoints [#endpoints]

<PlatformEndpoints platform="tavily" />

Response Shape [#response-shape]

Tavily endpoints use the `Analytics` archetype, which means the upstream response passes through with only the standard envelope-key strip applied. You see Tavily's full payload — including the `answer` field for search, the `failed_results` array for extract, and the `instructions` echo for crawl — exactly as Tavily returns it.

This differs from the social platforms (TikTok, Instagram, etc.) which apply a unified `Author` / `Post` field map. Web search results aren't social posts and aren't shoehorned into that schema.

Notes [#notes]

* All endpoints use `GET` method with query parameters
* Authentication via `x-api-key` header (your SocialCrawl key, not your Tavily key)
* Array-typed Tavily params (e.g. `urls`, `include_domains`, `select_paths`) are accepted as comma-separated strings on this surface and split server-side
* Empty search results (`results: []`) are billed normally — you paid for the search, an empty result is a real answer
* Caching: search responses cached 2 min, extract 10 min, map / crawl 30 min