# Tavily Crawl API (https://www.socialcrawl.dev/platforms/tavily/crawl)

> Multi-page crawl starting from a root URL. Returns each crawled page with its extracted content (unlike map, which returns only URLs). Use `instructions` to guide the crawler in natural language — Tavily uses an LLM to follow only the paths matching your intent. Use `select_*` / `exclude_*` filters (comma-separated regex patterns) to constrain scope.

TL;DR: `GET /v1/tavily/crawl` costs 1 credit per call and returns SocialCrawl's unified JSON schema. Single x-api-key auth, 100 free credits on signup.

## Parameters

| Parameter | Required | Description |
| --- | --- | --- |
| `url` | yes | Root URL to begin crawling. |
| `max_depth` | no | Maximum link depth from the root URL. Defaults to 1. |
| `max_breadth` | no | Maximum number of links followed per level (per page). Defaults to 20. |
| `limit` | no | Total number of pages the crawler will process before stopping. Defaults to 50. |
| `instructions` | no | Natural-language instructions for the crawler (e.g. 'Find all product pages with pricing'). |
| `select_paths` | no | Comma-separated regex patterns — only crawl URLs whose path matches. |
| `select_domains` | no | Comma-separated regex patterns — only crawl URLs whose domain matches. |
| `exclude_paths` | no | Comma-separated regex patterns — skip URLs whose path matches. |
| `exclude_domains` | no | Comma-separated regex patterns — skip URLs whose domain matches. |
| `allow_external` | no | Whether to follow links to external domains. Defaults to true. |
| `extract_depth` | no | Per-page extraction strategy. `basic` is faster; `advanced` handles harder pages. (basic \| advanced) |
| `format` | no | Output format for extracted content. `markdown` (default) or `text`. (markdown \| text) |
| `categories` | no | Comma-separated list of category hints to bias the crawl toward. |

## Code example

```bash
curl "https://www.socialcrawl.dev/v1/tavily/crawl?url=https%3A%2F%2Fdocs.tavily.com" \
  -H "x-api-key: sc_YOUR_API_KEY"
```

## FAQ

### How do I crawl a website with the Tavily API?

Send a GET request to /v1/tavily/crawl with the site's root in the url parameter and your SocialCrawl x-api-key. Tavily walks the site following links and returns the pages it visits with their extracted content — useful for ingesting docs, blogs, or knowledge bases.

### How does LLM-driven path selection work?

Pass natural-language instructions and Tavily uses an LLM to decide which links to follow toward your goal, instead of blindly crawling everything. For example, instruct it to follow only pricing and product pages, and it prioritises those paths.

### What parameters control the crawl?

instructions for natural-language guidance, max_depth for how many hops from the root, limit to cap pages, path and domain filters to scope the crawl, and extract options that control how each page's content comes back.

### When should I use crawl instead of map or search?

Use crawl when you need the actual content of many pages on a site, guided toward a goal. Use map when you only need the link structure, and search when you are finding pages for a query across the whole web. All three run on the same SocialCrawl key.

### How much does the Tavily Crawl API cost?

Each crawl call costs 1 credit on the standard tier. New accounts get 100 free credits with no credit card required — enough to crawl several sites and tune instructions, depth, and filters before you pay anything.

See the full Tavily API: https://www.socialcrawl.dev/platforms/tavily

## Pricing

- Standard endpoints: 1 credit per call
- Advanced endpoints: 5 credits per call
- Premium endpoints: 10 credits per call
- 100 free credits on signup, no credit card required. Cached responses cost 0 credits. Credit packs never expire.
- Full pricing: https://www.socialcrawl.dev/pricing

## Explore with AI

Questions this API answers, phrased for an AI assistant:

- How to crawl a website with LLM-driven path selection on SocialCrawl
- What is the difference between the Tavily crawl and map endpoints?