AffilyticsBot
Last updated: July 2026
AffilyticsBot is the crawler behind Affilytics. If you found this page, you probably saw it in your server logs and want to know what it is and how to control it. Short version: it visits a site only after that site's owner adds it to Affilytics to check their affiliate links, it obeys robots.txt, and you can block it in one line.
What it does
Affilytics helps creators find broken affiliate links across their content. When someone adds a website to their account, AffilyticsBot crawls that site's pages, collects the outbound links, and hands them off for health checking. That is the whole job.
It is not a search engine crawler, and it does not collect content for training. It reads the HTML your server returns and nothing else:
- It does not run JavaScript or render pages in a browser.
- It does not click affiliate links, submit forms, or log in.
- It does not crawl other subdomains or follow links off your site.
- It ignores private addresses and never tries to reach internal networks.
How to identify it
Every request carries this User-Agent:
AffilyticsBot/1.0 (+https://affilytics.io/bot)Match the AffilyticsBot token to write rules for it. Requests come directly from our servers, not through a residential or proxy network.
How to control it
AffilyticsBot reads your robots.txt before it crawls and follows it. To block it from the whole site:
User-agent: AffilyticsBot
Disallow: /To keep it out of one section, point the Disallow at that path:
User-agent: AffilyticsBot
Disallow: /members/It also honors Crawl-delay (capped at 10 seconds so a large value can't stall a scan). A full Disallow: / stops the scan with an error rather than reporting an empty site, so the account owner sees why nothing came back.
If your own scan comes back empty or blocked
If you added your own site to Affilytics and the scan found nothing, or the dashboard says it was blocked, something on your end is turning AffilyticsBot away before it can read your pages. Two things cause this.
Your robots.txt blocks it
Open https://yoursite.com/robots.txt. A Disallow: / under User-agent: * applies to us too. Add a group that names AffilyticsBot and allows it. The more specific group wins, so this lets us in without changing how any other bot is treated:
User-agent: AffilyticsBot
Allow: /Save it, then rescan the site from your dashboard.
A firewall or CDN blocks it
If robots.txt looks fine but pages still come back blocked, a bot filter such as Cloudflare is stopping the request before it reaches your site. Allow the AffilyticsBot user-agent in your firewall or bot-management rules. On Cloudflare that is a WAF rule that skips the bot check when User-Agent contains AffilyticsBot. If your firewall can only allow by IP address, email us and we'll send you the address we crawl from.
After either fix, trigger a fresh scan. Still empty? Send us the domain and we'll read the logs from our side.
How it treats your server
The crawl is meant to be light and to finish quickly:
- At most 2 requests in flight at once, spaced at least half a second apart.
- It prefers your sitemap, so most sites are covered in a couple of minutes.
- It backs off when it sees a 429 or 503 and respects
Retry-After. - Each scan is bounded by a page budget and a hard time limit, then it stops.
Still seeing a problem
If AffilyticsBot is hitting your site harder than it should, or you want it gone and a robots.txt rule isn't doing the job, email team@affilytics.io with your domain and a few log lines. We'll sort it out.
AffilyticsBot only crawls sites that an Affilytics account owner has added to check their own affiliate links.