# Tavily (/docs/tavily)

Web grounding via the Tavily API. Search, extract, map, and crawl

Search the current public web with optional LLM-generated answers, extract clean content from URLs, and crawl or map sitegraphs. Every endpoint is a `GET` and costs 1 credit.

Base URL: `/v1/tavily/...`

  Tavily endpoints use the `Analytics` archetype: the upstream response passes
  through with only the standard envelope-key strip applied. You see Tavily's
  full payload rather than the unified `Author` / `Post` field map that the
  social platforms use. Web search results are not social posts.

## Quickstart

### 1. Run a web search

```bash title="cURL"
curl "https://www.socialcrawl.dev/v1/tavily/search?query=claude+opus+release+notes" \
  -H "x-api-key: YOUR_API_KEY"
```

### 2. Ask for a written answer alongside the results

```bash title="cURL"
curl "https://www.socialcrawl.dev/v1/tavily/search?query=who+founded+anthropic&include_answer=true" \
  -H "x-api-key: YOUR_API_KEY"
```

The response includes both ranked `results[]` and a synthesised `answer` string.

## Search and extract

| Endpoint                 | Credits | What it returns                                                                        | Key parameters                                                                                                                                             |
| ------------------------ | ------- | -------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GET /v1/tavily/search`  | 1       | Ranked web results, plus an optional LLM-generated `answer` string                     | `query`, `search_depth`, `topic`, `time_range`, `max_results`, `include_answer`, `include_domains`, `exclude_domains`, `country`, `start_date`, `end_date` |
| `GET /v1/tavily/extract` | 1       | Clean extracted content for one or more URLs, with unreadable ones in `failed_results` | `urls` (up to 20, comma-separated), `extract_depth`, `format`, `include_images`, `timeout`                                                                 |

Any URL that could not be read comes back in `failed_results` rather than failing the whole request.

```bash title="cURL"
curl "https://www.socialcrawl.dev/v1/tavily/extract?urls=https://en.wikipedia.org/wiki/Lionel_Messi" \
  -H "x-api-key: YOUR_API_KEY"
```

## Walk a site

| Endpoint               | Credits | What it returns                                               | Key parameters                                                                                                                                                   |
| ---------------------- | ------- | ------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GET /v1/tavily/map`   | 1       | The URLs a site exposes, without any page bodies              | `url`, `max_depth`, `max_breadth`, `limit`, `instructions`, `select_paths`, `select_domains`, `exclude_paths`, `exclude_domains`, `allow_external`, `categories` |
| `GET /v1/tavily/crawl` | 1       | Each followed page's extracted content, starting from one URL | The same shaping parameters, plus `extract_depth` and `format`                                                                                                   |

Both share the same scoping parameters, so a map and the crawl that follows it can take identical settings. What separates them is `instructions`: plain-language guidance on which paths are worth following, which is Tavily's distinguishing feature over a mechanical link walk.

Use `map` when you want the shape of a site, and `crawl` when you want its text. Mapping first is the cheap way to find out what exists before you spend anything reading it.

```bash title="cURL"
# What pages does this site have?
curl "https://www.socialcrawl.dev/v1/tavily/map?url=https://docs.anthropic.com&max_depth=2&limit=100" \
  -H "x-api-key: YOUR_API_KEY"

# Read the ones about pricing
curl "https://www.socialcrawl.dev/v1/tavily/crawl?url=https://docs.anthropic.com&instructions=pages%20about%20pricing%20and%20rate%20limits&limit=20" \
  -H "x-api-key: YOUR_API_KEY"
```

## All endpoints

## Platform notes

- **Empty search results (`results: []`) are billed normally.** You paid for the search, and an empty result is a real answer.
- **Array-typed Tavily params** (`urls`, `include_domains`, `select_paths`, and the rest) are accepted as comma-separated strings on this surface and split server-side.
- Authentication is your SocialCrawl `x-api-key`, not a Tavily key.
- Caching: search responses cache for 2 minutes, extract for 10 minutes, map and crawl for 30 minutes. Cache hits cost 0 credits, and `Cache-Control: no-cache` requests a billed refresh.

## Next steps

- [Web scraping](/docs/web.md): The stateful surface: async crawls, monitors, and browser sessions.
- [Perplexity](/docs/perplexity.md): One synthesised answer with citations, for a question in words.
- [Caching](/docs/caching.md): TTLs, cache hits, and how to force a fresh fetch.