Contact crawler

SocialCrawlContactBot is the crawler SocialCrawl uses to find a contact email address that a creator publishes on their own website. To block it, add this to the site's robots.txt:

User-agent: SocialCrawlContactBot
Disallow: /

It identifies itself with this User-Agent on every request:

SocialCrawlContactBot/1.0 (+https://www.socialcrawl.dev/docs/contact-crawler; socialcrawl@ridio.io)

What does it read?

It reads a website only when a creator links that website from their own channel. It never guesses a site from a name or a handle.

On that site it reads at most two pages:

1. The home page the channel links to.
2. One more page on the same site, chosen from the home page's own links: the one whose text or path says contact, business, press, collaboration, work with me, or about.

It looks for a contact email address the creator published on those pages. It does not follow any other link, submit forms, log in, run scripts, or read anything behind a login.

How does it treat robots.txt?

It follows RFC 9309, the robots.txt standard, with the product token SocialCrawlContactBot.

• Rules in a group that names SocialCrawlContactBot apply. When no group names it, the User-agent: * rules apply.
• Within those rules, the longest matching Allow or Disallow path wins. On a tie, Allow wins.
• It reads robots.txt once per site and keeps the answer for up to 24 hours. A change to the file takes effect within that time.
• A robots.txt that answers 404, or another 4xx status other than 429, means the site has no rules for it, so it may read the two pages above.
• A robots.txt that answers a 5xx status or 429, times out, or cannot be reached is treated as a full disallow for 24 hours. The crawler reads nothing from the site in that time.

To keep it off one page only, disallow that path:

User-agent: SocialCrawlContactBot
Disallow: /contact

How fast does it go?

• At most one request to a domain every 2 seconds, robots.txt included.
• Each request stops after 5 seconds.
• It reads at most 1 MB of a page, and only text/html pages.
• It follows at most 3 redirects for a page and 5 for robots.txt.
• It sends no cookies and no credentials, and fetches only http and https on ports 80 and 443.

How do I opt out?

• Block the crawler: add the Disallow: / group at the top of this page to your robots.txt. This works without contacting us.
• Ask us: write to socialcrawl@ridio.io with the domain or the address concerned.

Next steps