A web crawler is an automated HTTP client that discovers URLs, fetches them on a schedule, and hands the content to a system that indexes it, trains on it, or answers questions with it. Googlebot, Bingbot, GPTBot and ClaudeBot are all crawlers. They differ in who runs them, how often they return, and whether they obey robots.txt, the Robots Exclusion Protocol standardized as RFC 9309 in September 2022.
Googlebot, the largest search crawler, reads only the first 500 KiB of a robots.txt file and the first 15 MB of each HTML or text file it fetches, according to Google Search Central documentation current in 2026. Anything past those limits is ignored, so an oversized robots.txt silently drops its trailing rules and an oversized page loses its trailing content.
A crawler runs a loop over a crawl frontier: a prioritized queue of URLs seeded from sitemaps, previously known URLs and links extracted from fetched pages. Each host gets its own politeness budget, and the scheduler decides which URL to fetch next based on how important the page looks and how often it has changed in the past.
Before fetching any URL on a host, a well-behaved crawler requests robots.txt and caches it. Google generally caches it for up to 24 hours. A 4xx on robots.txt is read as "no restrictions"; a 5xx makes Googlebot pause crawling the site, because it cannot tell what is allowed.
The fetch itself is ordinary HTTP. Googlebot can crawl over HTTP/2, sends conditional requests with If-None-Match and If-Modified-Since when it holds an ETag or Last-Modified value, and accepts a 304 Not Modified with no body. The response is parsed for links, canonicals and directives, deduplicated, and for search engines optionally queued for JavaScript rendering, which happens later and separately from the raw fetch.
Crawl rate is adaptive. Google documents that Googlebot slows down when it sees 500, 503 or 429 responses and speeds up when the host answers quickly. Google dropped support for the crawl-delay directive and retired the Search Console crawl rate limiter in January 2024, so server responses are now the only throttle you control for Googlebot.
Crawler requests land on the same edge as users, but they behave like the worst possible client for a cache. Humans concentrate on a hot set of URLs; a crawler walks the long tail, so a large share of its requests are cache misses that go back to origin or origin shield.
That makes crawl traffic an origin-load and egress problem more than an edge-capacity one. Separate it in your logs by user agent, then confirm identity: a user agent string is trivially spoofed. Google's documented check is a reverse DNS lookup on the client IP that resolves into googlebot.com or google.com, followed by a forward lookup that returns the same IP. Google, OpenAI and several other operators also publish their crawler IP ranges as machine-readable lists.
An AI crawler fetches pages to train language models, to build a retrieval index for AI answers, or to read a page on demand because a user asked an assistant about it. Search crawlers send referral traffic back in exchange for the fetch; training crawlers send none, which is why sites now treat the categories differently.
Operators increasingly split these roles into separate user agents. OpenAI documents GPTBot for training, OAI-SearchBot for search and ChatGPT-User for user-initiated fetches. Anthropic documents ClaudeBot, Claude-SearchBot and Claude-User. Perplexity runs PerplexityBot and Perplexity-User. Vendors note that user-initiated fetchers can be governed by different rules than their training crawlers.
Google-Extended is the exception that confuses most teams: it is a robots.txt product token, not a separate crawler. Googlebot still does the fetching; disallowing Google-Extended only controls whether that content may be used for Gemini model training and grounding, and it has no effect on Search indexing. Applebot-Extended works the same way for Apple. Public traffic measurements also show that many AI crawlers fetch raw HTML without executing JavaScript, so content rendered only client-side may be invisible to them.
Web crawler vs. scraper: a crawler is defined by discovery, following links across many pages and hosts on a recurring schedule. A scraper extracts specific fields from known pages. Many scrapers crawl, and many crawlers never extract structured data, but robots.txt etiquette is only reliably followed by declared crawlers.
Web crawler vs. indexer: the crawler fetches bytes; the indexer parses, ranks and stores them. A page can be crawled and never indexed, and a URL blocked from crawling can still be indexed from external links, appearing in results without a snippet.
Web crawler vs. user-triggered fetcher: a crawler decides what to fetch on its own schedule. A fetcher such as ChatGPT-User or Claude-User requests one URL because a person asked, so its volume tracks user demand rather than your site's size.
Crawl bytes scale with file size, not page count. The table below is this blog's arithmetic for three illustrative workloads, assuming six active crawlers (two search, four AI) each completing the stated passes per month in decimal units.
| Workload | Objects crawled | Average transfer | Passes per month | Per crawler | Six crawlers |
|---|---|---|---|---|---|
| Content site, HTML only | 2,000,000 pages | 120 KB compressed | 1 | 240 GB | 1.44 TB |
| Documentation with PDFs | 50,000 PDFs | 4 MB | 2 | 400 GB | 2.4 TB |
| Software downloads, links not disallowed | 10,000 installers | 300 MB | 1 | 3 TB | 18 TB |
Two million crawled pages cost less than 10,000 crawlable binaries, so the highest-leverage rule is keeping crawlers away from large files, not limiting page count.
The 18 TB case is avoidable with one robots.txt line per download path, and 304 responses cut repeat passes on unchanged HTML to headers only. If the remaining crawl bytes run through a CDN, BlazingCDN bills per TB with no request fees and includes rate limiting, IP lists and referrer rules on every plan (see its CDN access-control and delivery features), though it is a delivery layer and not a bot-management product.
A common 2026 pattern, described in words: one group for User-agent Googlebot and Bingbot that allows the site but disallows the downloads and internal search paths; one group for GPTBot, ClaudeBot, CCBot and the Google-Extended token with Disallow set to the root path, if you opt out of training; a separate decision for OAI-SearchBot, Claude-SearchBot and PerplexityBot, which drive AI search citations; and a Sitemap line with the absolute sitemap URL. Rules in RFC 9309 match by the most specific user-agent group, so a named group fully replaces the wildcard group for that crawler.
For non-HTML files that may be fetched but should not appear in results, send the X-Robots-Tag response header with the value noindex. For rate pressure, return 503 or 429 with a Retry-After header briefly, never 403 or 404, which Google treats as content signals rather than load signals.
A web crawler also feeds AI model training, AI search retrieval indexes, SEO audit tools, price monitoring, archiving and link checking. The fetch mechanics are identical across uses; what differs is the declared user agent, the recrawl frequency and whether the operator publishes IP ranges and honors robots.txt. Classify crawlers by purpose before deciding which ones your robots.txt and rate limits should allow.
Add a robots.txt group naming the AI crawler's user agent, such as GPTBot or ClaudeBot, with a disallow on the root path, and leave Googlebot's group unchanged. For Gemini training, disallow the Google-Extended token, which does not affect Search indexing. Because robots.txt is voluntary, pair it with verified IP lists and edge rate limits for crawlers that ignore it.
Major declared crawlers from Google, Microsoft, OpenAI, Anthropic and Apple document that they honor robots.txt under RFC 9309, but compliance is voluntary. Scrapers and spoofed user agents often ignore it, and user-triggered fetchers may follow different rules. Verify claimed crawler identity with reverse and forward DNS lookups or the operator's published IP ranges before trusting any user agent string.
Web crawler bandwidth depends on the bytes of crawlable objects, not page count. Two million 120 KB pages crawled once by six bots is about 1.44 TB a month, while 10,000 crawlable 300 MB installers reach 18 TB. Disallowing binary download paths and answering conditional requests with 304 Not Modified usually removes most avoidable crawl transfer.
Pull 30 days of edge or origin logs and group bytes sent by user agent, then by path prefix. Verify the top ten agents with reverse DNS or published IP ranges and tag anything unverifiable as spoofed. Check three numbers: crawler share of total bytes, crawler cache-miss ratio against human traffic, and the share of crawler requests answered with 304. If large binaries show up under any crawler, a single robots.txt disallow on that path is the cheapest bandwidth fix you will ship this quarter.
If the crawl bytes that remain are a meaningful slice of your egress, the BlazingCDN CDN cost calculator prices that traffic on progressive tiers from $5 per TB down to $2.50 per TB at volume, with no request fees, as of November 2026.