What Are Bots? Good vs Bad Bots & Bot Traffic Explained (2026)

A bot is an automated software agent that issues HTTP requests without a human driving each action, running on a schedule or in response to events rather than clicks. In 2024–2025 public traffic measurements, bots generated roughly 47–50% of all web requests. Some are essential (search crawlers, uptime monitors, payment webhooks); others scrape content, hoard inventory, or brute-force credentials. Understanding bot traffic means separating declared, well-behaved automation from traffic that disguises itself as a browser.

What is a bot, and how does bot traffic actually work?

A bot is a program that speaks HTTP directly. It opens a connection, sends a request line and headers, reads the response, and repeats, often thousands of times per minute from a single process. The defining trait is not the payload but the driver: code, not a person.

Most legitimate bots declare themselves in the User-Agent header and honor robots.txt. A crawler like Googlebot fetches your sitemap, respects Crawl-delay semantics where implemented, and issues conditional requests with If-Modified-Since to avoid re-downloading unchanged assets. Malicious bot traffic does the opposite: it spoofs a Chrome User-Agent, ignores robots.txt, rotates source IPs, and often runs a full headless browser to execute JavaScript and defeat naive filtering.

At scale this matters because every request costs something. A scraper hammering uncached product pages at 200 requests per second bypasses your edge cache, lands on origin, and burns database connections and egress that you pay for whether the traffic is human or not.

Good bots vs bad bots: how to tell them apart

The good bots vs bad bots distinction comes down to declaration, verifiability, and rate. Good bots identify themselves honestly and can be confirmed; bad bots lie about identity and behave abusively.

• Good bots — search crawlers (Googlebot, Bingbot), AI training and retrieval crawlers (GPTBot, ClaudeBot), uptime monitors, link previewers, and payment webhooks. They declare a stable User-Agent and usually publish an IP range or support reverse-DNS verification.
• Bad bots — content scrapers, price and inventory scrapers, credential-stuffing scripts, comment spammers, and fake-account creators. They forge headers, distribute across residential proxies, and pace requests to evade simple rate limits.

The trap: a spoofed User-Agent proves nothing. Verify a claimed Googlebot with a reverse DNS lookup on the source IP, then a forward lookup back to the same address. If they don't match, the "crawler" is lying.

Where bots sit in the stack, and how a CDN handles them

Bot traffic hits your edge first, which makes the CDN layer the correct place to classify and shape it. Cacheable requests from well-behaved web crawlers can be served entirely from cache, never touching origin. Requests that miss cache or target dynamic endpoints are where cost and risk concentrate.

A minimal edge classification looks like this in nginx terms:

map $http_user_agent $is_known_bot {
    default        0;
    "~*Googlebot"  1;
    "~*bingbot"    1;
    "~*GPTBot"     1;
    "~*ClaudeBot"  1;
}

# Rate-limit unverified automated traffic separately from humans
limit_req_zone $binary_remote_addr zone=bots:10m rate=5r/s;

This is a starting point, not a defense: the User-Agent match trusts a string anyone can forge, which is why verification and per-IP rate policy sit alongside it. Offloading crawler-heavy read traffic to a high-cache-hit edge keeps origin load flat even when aggregate bot traffic doubles. A CDN with flexible edge caching and request-rate controls lets you absorb crawler surges at the edge instead of paying for them at origin.

Bot vs. neighboring terms

Bot vs. web crawler: A crawler is a specific kind of bot that discovers and indexes content by following links. All crawlers are bots; most bots (webhooks, monitors, scripts) are not crawlers.

Bot vs. scraper: A scraper extracts and stores data from pages, often ignoring robots.txt. Crawlers index for search; scrapers copy for reuse. The overlap in mechanics is why scrapers frequently impersonate legitimate crawlers.

Bot vs. headless browser: A headless browser (Chromium without a UI) is a tool, not a category. Good bots and bad bots both use it; the headless browser just makes bad bots harder to distinguish from real users because it executes JavaScript and renders like Chrome.

Common misconceptions about bot traffic

"Blocking the User-Agent stops the bot." No. Abusive bots rotate User-Agent strings and IPs freely; identity strings are advisory, not enforcement.

"All bots hurt performance." No. Search and AI crawlers drive discovery and traffic. The goal is to shape and verify automated traffic, not eliminate it.

"Bot traffic is a small fraction of load." No. In 2025 measurements, automated requests approached half of all traffic, and on content-heavy sites crawlers alone can exceed human page views.

FAQ: bots and bot traffic explained

What percentage of web traffic is bots in 2026?

Automated bots account for roughly 47–50% of global web requests in 2024–2025 public measurements, and that share is expected to hold or rise through 2026 as AI retrieval crawlers grow. The split between good and bad bots varies by site, but on content-heavy properties crawlers frequently generate more requests than human visitors.

How do I verify that a bot is really Googlebot?

Run a reverse DNS lookup on the request's source IP, confirm it resolves to a googlebot.com or google.com hostname, then run a forward DNS lookup on that hostname and check it returns the original IP. A matching round trip confirms the crawler; a mismatch means the User-Agent is spoofed.

Do bots increase CDN and origin costs?

Yes, when their requests miss cache. Cacheable crawler requests served from the edge add negligible origin cost, but scrapers targeting dynamic or uncached endpoints consume origin compute, database connections, and egress you pay for. Raising cache hit ratio and rate-limiting unverified automated traffic are the two highest-leverage cost controls.

Should I block AI crawlers like GPTBot?

It depends on your content strategy. Blocking GPTBot or ClaudeBot via robots.txt removes your pages from that model's retrieval, which reduces AI-answer visibility. Many sites allow retrieval crawlers for discovery while rate-limiting them so a single crawler cannot dominate origin capacity during a re-index.

Instrument your bot traffic this week

Pull one day of access logs and bucket requests three ways: verified good bots (round-trip DNS confirmed), unverified automation (bot-like User-Agent, no verification), and humans. Then cross-reference each bucket against cache status. If unverified automation is landing on origin at a meaningful rate, you've found free money: cache those paths or rate-limit the offenders. Want a second data point? Compare your crawler request volume to human page views. If crawlers win, your caching strategy, not your app, is your performance ceiling.