Good bots explained: how to recognize them, robots.txt, and why you still need visibility

Good bots are automated clients you often want on your site: search crawlers, uptime monitors, payment or partner integrations, and some AI assistants that fetch pages with clear provenance. The challenge is telling them apart from scrapers and abuse — and governing both with policy, not vibes.

If you run a European media site, marketplace, or SaaS, good-bot traffic affects SEO, availability, and infrastructure cost. Ignoring it is as risky as blocking it blindly.

What counts as a “good” bot?

“Good” is a business judgment, not a moral label. Typical categories:

For deeper product context, see crawler management and scraper protection.

How to recognize good bots (beyond the User-Agent)

1. Declared identity (User-Agent)

Cooperative crawlers usually identify themselves. Examples you will see in logs (strings evolve — treat these as patterns, not a forever allowlist):

# Search
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; DuckDuckBot-Https/1.1; https://duckduckgo.com/duckduckbot)
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)

# Previews / unfurls
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
Twitterbot/1.0
LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com)

# Monitoring (vendor-specific; examples)
Mozilla/5.0 (compatible; Pingdom.com_bot_version_check)
Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)

# AI / assistant crawlers (policies differ by vendor — verify current docs)
Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com)

Important: Anyone can set User-Agent: Googlebot. Recognition requires verification, not string matching alone.

2. Network verification

For major search crawlers, the standard approach is:

1. Reverse DNS lookup on the connecting IP
2. Confirm the hostname belongs to the crawler’s domain (e.g. *.googlebot.com)
3. Forward-confirm DNS back to the same IP

If that chain fails, treat the client as spoofed — even if the User-Agent looks perfect.

3. Behavior on your routes

Good bots tend to:

• Fetch publicly linked pages and assets at a steady pace
• Honor (or at least attempt) crawl-delay / robots rules when they claim to
• Avoid credential stuffing patterns on /login and /signup
• Show consistent ASN / IP ownership over time

Abusive automation often concentrates on inventory, pricing APIs, forms, or account endpoints — see our primer on what bot traffic is.

robots.txt: still worth doing (even when some ignore it)

robots.txt is a voluntary convention at your site root (https://example.com/robots.txt). Cooperative crawlers read it; many scrapers and custom scripts do not.

Use it anyway:

• It documents intent for legitimate crawlers and AI bots that honor it
• It reduces accidental overcrawl of staging paths, faceted URLs, and thin parameters
• It is not access control — sensitive data needs auth, not a Disallow line

Minimal example

# https://www.example.com/robots.txt
User-agent: *
Disallow: /admin/
Disallow: /api/internal/
Disallow: /cart
Disallow: /*?*sort=
Allow: /

# Example: be explicit for a named crawler (check that vendor’s current product name)
User-agent: GPTBot
Disallow: /private/
Allow: /blog/

Sitemap: https://www.example.com/sitemap.xml

Tips that age well:

• Keep one canonical robots.txt on the production host
• Point to your XML sitemap
• Prefer specific rules over a blanket Disallow: / unless you truly want to de-index
• Revisit AI crawler tokens as vendors rename agents — policies change faster than search bots

When a client ignores robots.txt, that is a classification signal for bot protection: cooperative vs non-cooperative automation.

Why a bot management tool should analyze good bots too

Allowlisting “Googlebot” in a WAF and forgetting the rest leaves blind spots. A proper Bot Shield-style control plane helps because good bots still:

1. Cost money — Crawl volume drives CDN, origin CPU, and cache churn
2. Shape SEO and previews — Mis-challenging a real crawler hurts indexing and unfurls
3. Overlap with scrapers — Same paths, similar concurrency; you need labels, not just blocks
4. Change over time — New AI agents appear; verification rules and allowlists need updates
5. Inform policy — You may allow Googlebot on /, throttle aggressive AI crawl on /pricing, and block spoofed UAs on /login

Without analytics that separate good, bad, and unknown automation, teams either over-block (false positives) or under-protect (silent scrape and abuse). Visibility turns robots.txt and User-Agent folklore into measurable policy: allow, challenge, or block per route.

A practical setup checklist

1. Publish a clear robots.txt + sitemap; document AI crawler intent
2. Verify major crawlers by DNS/IP — never by User-Agent alone
3. Allowlist known monitors and partners on health-check paths only
4. Keep login, signup, and payment routes on stricter sensitivity
5. Review bot reports weekly: volume by class, robots.txt violators, spoofed UAs
6. Prefer adaptive friction over permanent puzzles for ambiguous traffic (CAPTCHA alternative)

Bottom line

Good bots are real, useful, and sometimes expensive. Recognize them with identity + verification + behavior, publish robots.txt for cooperative clients (knowing others will ignore it), and use bot management to analyze and govern legitimate automation — not only to stop the bad kind.

Want help classifying crawlers on your stack? Book a demo or request a free bot analysis.