# Contact crawler (/docs/contact-crawler)

What SocialCrawlContactBot reads on a creator's website, how it honors robots.txt, how fast it goes, and how to opt out

SocialCrawlContactBot is the crawler SocialCrawl uses to find a contact email address that a creator publishes on their own website. To block it, add this to the site's robots.txt:

```text title="robots.txt"
User-agent: SocialCrawlContactBot
Disallow: /
```

It identifies itself with this User-Agent on every request:

```text title="User-Agent"
SocialCrawlContactBot/1.0 (+https://www.socialcrawl.dev/docs/contact-crawler; socialcrawl@ridio.io)
```

## What does it read?

It reads a website only when a creator links that website from their own channel. It never guesses a site from a name or a handle.

On that site it reads at most two pages:

1. The home page the channel links to.
2. One more page on the same site, chosen from the home page's own links: the one whose text or path says contact, business, press, collaboration, work with me, or about.

It looks for a contact email address the creator published on those pages. It does not follow any other link, submit forms, log in, run scripts, or read anything behind a login.

## How does it treat robots.txt?

It follows [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309), the robots.txt standard, with the product token `SocialCrawlContactBot`.

- Rules in a group that names `SocialCrawlContactBot` apply. When no group names it, the `User-agent: *` rules apply.
- Within those rules, the longest matching `Allow` or `Disallow` path wins. On a tie, `Allow` wins.
- It reads robots.txt once per site and keeps the answer for up to 24 hours. A change to the file takes effect within that time.
- A robots.txt that answers 404, or another 4xx status other than 429, means the site has no rules for it, so it may read the two pages above.
- A robots.txt that answers a 5xx status or 429, times out, or cannot be reached is treated as a full disallow for 24 hours. The crawler reads nothing from the site in that time.

To keep it off one page only, disallow that path:

```text title="robots.txt"
User-agent: SocialCrawlContactBot
Disallow: /contact
```

## How fast does it go?

- At most one request to a domain every 2 seconds, robots.txt included.
- Each request stops after 5 seconds.
- It reads at most 1 MB of a page, and only `text/html` pages.
- It follows at most 3 redirects for a page and 5 for robots.txt.
- It sends no cookies and no credentials, and fetches only `http` and `https` on ports 80 and 443.

## How do I opt out?

- **Block the crawler:** add the `Disallow: /` group at the top of this page to your robots.txt. This works without contacting us.
- **Ask us:** write to socialcrawl@ridio.io with the domain or the address concerned.

## Next steps

- [Security](/docs/security.md): How SocialCrawl handles requests, public data and stored records.
- [Public Data Notice](/legal/public-data-notice): The public-source collection boundary and your rights.