---
title: "AI Bots and Your Bandwidth Bill: Measuring AI Crawler Traffic"
description: Measure AI crawler traffic, see how AI bots consume bandwidth, and use robots.txt AI rules to control crawler access and costs.
image: https://blog.blazingcdn.com/hubfs/Gemini-Blog/image-Nov-19-2026-03-08-22-6265-PM.jpeg
---

[![BlazingCDN](https://blog.blazingcdn.com/hubfs/Logo/blog-logo-w.png)](https://blog.blazingcdn.com?hsLang=en-us)

[📘 Learn ▾](https://blog.blazingcdn.com/cdn-learn?hsLang=en-us)

[CDN Fundamentals](https://blog.blazingcdn.com/cdn-fundamentals?hsLang=en-us) [By Content Type](https://blog.blazingcdn.com/by-content-type?hsLang=en-us) [Advanced Concepts](https://blog.blazingcdn.com/advanced-concepts?hsLang=en-us) [📖 Glossary](https://blog.blazingcdn.com/glossary?hsLang=en-us)

[🎬 Video & Streaming ▾](https://blog.blazingcdn.com/video-streaming-cdn?hsLang=en-us)

[🔴 Live Streaming](https://blog.blazingcdn.com/live-streaming?hsLang=en-us) [📺 VOD & OTT](https://blog.blazingcdn.com/vod-ott?hsLang=en-us) [🎞️ Encoding & Players](https://blog.blazingcdn.com/encoding-players?hsLang=en-us) [🎧 Audio & Podcasts](https://blog.blazingcdn.com/audio-podcasts?hsLang=en-us) [💰 Bandwidth & Costs](https://blog.blazingcdn.com/bandwidth-costs?hsLang=en-us)

[🏗️ Infrastructure ▾](https://blog.blazingcdn.com/cdn-infrastructure?hsLang=en-us)

[🗄️ Storage & Origin](https://blog.blazingcdn.com/storage-origin?hsLang=en-us) [📦 Large File Delivery](https://blog.blazingcdn.com/large-file-delivery?hsLang=en-us) [💸 Cloud Costs & Egress](https://blog.blazingcdn.com/cloud-costs-egress?hsLang=en-us) [🌐 Networking & ISP](https://blog.blazingcdn.com/networking-isp?hsLang=en-us) [⚙️ DevOps & Cloud Infra](https://blog.blazingcdn.com/devops-cloud-infra?hsLang=en-us) [⚡ Web Performance](https://blog.blazingcdn.com/web-performance?hsLang=en-us)

[🏭 Industries ▾](https://blog.blazingcdn.com/cdn-industry-insights?hsLang=en-us)

[📺 Media & Broadcasting](https://blog.blazingcdn.com/media-broadcasting?hsLang=en-us) [💾 Software & SaaS](https://blog.blazingcdn.com/software-saas?hsLang=en-us) [📣 AdTech & Advertising](https://blog.blazingcdn.com/adtech-advertising?hsLang=en-us) [🎮 Gaming & Esports](https://blog.blazingcdn.com/gaming-esports?hsLang=en-us) [📱 Mobile Apps & Developers](https://blog.blazingcdn.com/mobile-apps-developers?hsLang=en-us) [🤖 AI & Machine Learning](https://blog.blazingcdn.com/ai-machine-learning?hsLang=en-us) [🏛️ Enterprise & Corporate](https://blog.blazingcdn.com/enterprise-corporate?hsLang=en-us) [📚 E-Learning & EdTech](https://blog.blazingcdn.com/e-learning-edtech?hsLang=en-us) [🏟️ Sports & Live Events](https://blog.blazingcdn.com/sports-live-events?hsLang=en-us)

[💰 Pricing & Compare ▾](https://blog.blazingcdn.com/cdn-pricing-and-cdn-costs?hsLang=en-us)

[Provider Pricing](https://blog.blazingcdn.com/provider-pricing?hsLang=en-us) [Cost Optimization](https://blog.blazingcdn.com/cost-optimization?hsLang=en-us) [Decision Support](https://blog.blazingcdn.com/decision-support?hsLang=en-us) [⚡ Compare](https://blog.blazingcdn.com/cdn-comparison?hsLang=en-us) [Provider Comparisons](https://blog.blazingcdn.com/provider-comparisons?hsLang=en-us) [Strategy Comparisons](https://blog.blazingcdn.com/strategy-comparisons?hsLang=en-us) [📊 Benchmarks](https://blog.blazingcdn.com/cdn-ratings-and-benchmarks?hsLang=en-us)

[🔒 Security ▾](https://blog.blazingcdn.com/cdn-security?hsLang=en-us)

[Attack Protection](https://blog.blazingcdn.com/attack-protection?hsLang=en-us) [Encryption & Access](https://blog.blazingcdn.com/encryption-access?hsLang=en-us) [Video & DRM Security](https://blog.blazingcdn.com/video-drm-security?hsLang=en-us)

[🛠️ Tools](https://blog.blazingcdn.com/cdn-tools?hsLang=en-us)

[Get Started](https://blazingcdn.com/sign-up-contact-form/)

[Industries](https://blog.blazingcdn.com/en-us/tag/industries) [AI & Machine Learning](https://blog.blazingcdn.com/en-us/tag/ai-machine-learning) [Infrastructure](https://blog.blazingcdn.com/en-us/tag/infrastructure) [Infrastructure - Cloud Costs & Egress](https://blog.blazingcdn.com/en-us/tag/infrastructure-cloud-costs-egress)

# AI Bots and Your Bandwidth Bill: Measuring AI Crawler Traffic

 Vadym Borovskyi  Nov 19, 2026, 4:11:04 PM

![\<span id="hs\_cos\_wrapper\_name" class="hs\_cos\_wrapper hs\_cos\_wrapper\_meta\_field hs\_cos\_wrapper\_type\_text" style="" data-hs-cos-general-type="meta\_field" data-hs-cos-type="text" \>AI Bots and Your Bandwidth Bill: Measuring AI Crawler Traffic\</span\>](https://blog.blazingcdn.com/hubfs/Gemini-Blog/image-Nov-19-2026-03-08-22-6265-PM.jpeg)

AI bots are now a line item on the CDN invoice, and on file-heavy sites they can be the largest one. Read the Docs reported that a single AI crawler pulled 73 TB of zipped HTML in May 2024, almost 10 TB of it in one day, which cost the project more than $5,000 in bandwidth. This playbook takes about two hours of hands-on work. You will size AI crawler traffic in your CDN logs, put a monthly dollar figure on it, and cut it with robots.txt groups, cache rules and edge rate limits. Plan for 30 minutes on the first log pass and up to 24 hours for robots.txt changes to take effect.

AI crawler traffic costs more per byte than human traffic because crawlers walk the long tail, where cache hit ratios are low and most bytes come from origin. The Wikimedia Foundation reported in April 2025 that bots generated 65% of its most expensive traffic but only about 35% of its pageviews. It also reported that bandwidth for multimedia downloads had grown 50% since January 2024.

## **How much AI crawler traffic hits a typical CDN?**

There is no universal share. The answer depends almost entirely on bytes per fetch. HTML-only crawling of a content site usually stays in single-digit terabytes a month. A crawler that fetches images, archives or versioned downloads can move tens of terabytes, as the Read the Docs case shows. Measure your own bytes before you change any rules.

The model is simple. Monthly crawler bytes equal unique URLs, times fetches per URL per month, times crawler count, times average transfer size. Origin bytes equal that figure multiplied by the crawler miss ratio, which for long-tail URLs is often 30% to 60% (estimate). Human traffic on the same site may miss only 5%.

The table below applies that model to four scenarios at two assumed blended egress rates, $20 and $80 per TB. Neither rate is a vendor price. Rows are ordered by monthly bytes, lowest first.

| Scenario (assumptions) | Fetches per month | Bytes per month | At $20 per TB | At $80 per TB |
| --- | --- | --- | --- | --- |
| HTML only: 400,000 pages at 120 KB, 6 crawlers, 4 recrawls each | 9.6 million | 1.15 TB | $23 | $92 |
| Images: 1.5 million at 350 KB, 2 crawlers, 2 fetches each | 6 million | 2.1 TB | $42 | $168 |
| Archives: 2,000 files at 150 MB, 1 crawler refetching daily | 60,000 | 9 TB | $180 | $720 |
| Reported case: Read the Docs, one crawler, May 2024 | not published | 73 TB | $1,460 | $5,840 |

One crawler on large objects outweighs six crawlers on HTML, so rank your bot rules by bytes per fetch, not by request count.

## **Prerequisites**

- At least 7 days of raw CDN access logs, 30 if you have them. Each line needs timestamp, client IP, user agent, request path, status code, bytes sent and cache status.
- A place to aggregate them: a columnar warehouse, a log analytics service or a local DuckDB file.
- Write access to robots.txt at the origin and to the CDN or reverse proxy config (nginx examples below).
- Your effective egress rate per TB for both CDN delivery and origin transfer.

## **Step by step: identify AI bots in your CDN logs**

### Step 1: Build the token list

Match the user agent field case-insensitively against these tokens: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, CCBot, Bytespider, Amazonbot and Meta-ExternalAgent. Why: you are tagging rows, not blocking anything, so a broad list costs nothing at this stage.

### Step 2: Aggregate bytes, not hits

Group by token. For each one, sum bytes sent, count requests, compute the share of rows with a cache status of HIT, and list the top 20 paths by bytes. Why: request counts hide the archive crawler that makes 60,000 requests and moves 9 TB.

### Step 3: Verify the claimed identity

OpenAI and Perplexity publish IP range files for their crawlers. Join on those ranges and treat any GPTBot row from outside them as spoofed. For operators that document reverse DNS, resolve the client IP, then forward-confirm the hostname. Why: spoofed rows will not respond to robots.txt and need IP or rate rules instead.

### Step 4: Split crawlers by role

Tag each token as training, search indexing or user-triggered fetch. Why: blocking a training crawler saves bytes with no visible downside, while blocking OAI-SearchBot or Claude-SearchBot removes you from assistant answers and their referral clicks.

### Step 5: Price it

Multiply the total bot bytes by your CDN rate, then multiply the bot miss bytes by your origin rate. Report both figures. The origin figure is usually the one that justifies the engineering time.

Treat the result as a floor. Some scrapers present a browser user agent from residential IPs. A cluster of non-bot rows with zero referrers, no asset fetches after HTML and a steady request rate deserves a second look.

## **How to write robots.txt AI rules that actually save bandwidth**

Write robots.txt AI rules as separate groups per crawler token, and disallow heavy paths before you consider disallowing the whole site. Under RFC 9309, crawlers should not use a cached robots.txt for more than 24 hours, so expect a delay of up to a day. Compliance is voluntary, so you must verify the effect in logs.

The table lists the main tokens alphabetically by operator, with what a Disallow does to your egress.

| Operator | Tokens | Does a Disallow reduce bytes? |
| --- | --- | --- |
| Anthropic | ClaudeBot, Claude-SearchBot, Claude-User | Yes for ClaudeBot. Blocking the search and user agents also reduces answer visibility. |
| Apple | Applebot-Extended | No. It is a usage control, and Applebot still fetches. |
| ByteDance | Bytespider | Only if it complies. Verify in logs and back it with an edge rule. |
| Common Crawl | CCBot | Yes. Its corpus feeds many training sets downstream. |
| Google | Google-Extended | No. Googlebot does the fetching, and the token controls only Gemini usage. |
| Meta | Meta-ExternalAgent | Yes. |
| OpenAI | GPTBot, OAI-SearchBot, ChatGPT-User | Yes for GPTBot. OpenAI documents that user-initiated fetches may not apply robots.txt rules. |
| Perplexity | PerplexityBot, Perplexity-User | Yes for PerplexityBot. Perplexity-User is documented as generally ignoring robots.txt. |

Google-Extended and Applebot-Extended are permission signals, so adding them to robots.txt changes model training rights but saves zero bandwidth.

### Step 6: Disallow the heavy paths first

For each training token, add a group with **User-agent** set to the token, then a **Disallow** line for each path that sits in your top 20 by bytes, for example the downloads directory, the media directory and any versioned archive tree. Keep HTML open if you want to stay in training corpora. Why: this removes the 9 TB row from the table above while leaving the 1 TB row intact.

### Step 7: Make robots.txt itself reliable

Serve robots.txt from cache with a short TTL and make sure it never returns a 5xx. Under RFC 9309, a 5xx means "assume complete disallow", while a 404 means "allow everything". Why: an origin outage should not silently rewrite your crawl policy.

## **Caching rules that make the remaining AI bots cheap to serve**

### Step 8: Cache negative responses

Crawlers request dead URLs constantly. In nginx, set **proxy\_cache\_valid** for 404 responses to 10 minutes alongside your 200 rule. Why: every uncached 404 is an origin round trip.

### Step 9: Normalize the cache key

Build **proxy\_cache\_key** from scheme, host, URI and an allowlist of query arguments that actually change the response. Drop tracking and sort parameters. Never vary on User-Agent. Why: crawlers generate parameter permutations that otherwise fragment the cache and push the bot miss ratio toward 60%.

### Step 10: Coalesce and serve stale

Turn **proxy\_cache\_lock** on and set **proxy\_cache\_use\_stale** to include error, timeout and updating. Why: when a crawler sweeps thousands of cold URLs in parallel, only one request per object reaches origin.

### Step 11: Rate limit only the bots

Use a **map** on the http\_user\_agent variable that outputs the client address for listed tokens and an empty string for everything else. Feed that variable to **limit\_req\_zone** with a rate such as 5 requests per second, and set **limit\_req\_status** to 429. nginx does not account requests with an empty key, so human traffic is untouched. Why: a 429 is cheaper than a 150 MB 200, and well-behaved crawlers back off.

On a managed CDN, look for the same primitives: edge rate limiting, IP lists and per-user-agent stats. BlazingCDN includes rate limiting, geo and IP lists, real-time statistics and Prometheus metrics on every plan with no request fees, as listed on its [**CDN features page for rate limiting and IP lists**](https://blazingcdn.com/features/). It is a delivery layer, not a bot management product, so keep your existing security stack for fingerprinting.

## **Validate: which numbers should move**

- Within 48 hours, each compliant token should show a 200 on robots.txt, followed by near-zero bytes on disallowed paths.
- The bot cache HIT share should rise from Step 2's baseline. A target of 80% or more on the bot segment is reasonable for static sites (estimate).
- Origin bytes attributed to bot misses should fall by roughly the disallowed share of Step 2's top paths.
- The 429 count should come entirely from tagged tokens. If any 429 carries a browser user agent, your map default is wrong.

## **Rollback**

Version robots.txt in the same repository as your edge config, and redeploy the previous file if search or assistant referrals drop. Remember the 24-hour cache window. To roll back the rate limit, remove the **limit\_req** line and reload. The cache key change is the riskiest step, so ship it behind a separate cache zone and switch back by pointing the location at the old zone.

## **Failure modes: symptom, cause, fix**

- **Bytes for a token unchanged after Disallow.** Cause: spoofed user agent or a non-compliant crawler. Fix: check Step 3's IP verification, then add an edge IP list or rate rule.
- **Organic referrals drop.** Cause: a group accidentally matched Googlebot or a search agent such as OAI-SearchBot. Fix: audit group tokens and test with a robots.txt parser.
- **Bot hit ratio stays below 50%.** Cause: cache key fragmentation or short TTLs on long-tail HTML. Fix: tighten the argument allowlist and raise TTLs on immutable versioned paths.
- **Origin load spikes during crawls.** Cause: request coalescing is off or the cache lock timeout is too short. Fix: enable the lock and add origin shield.

## **Tuning once it works**

Schedule Step 2 as a weekly job and alert when any single token exceeds 10% of monthly egress or doubles week over week. Tokens change, so review the list each quarter against operator documentation. For large files, signed URLs on download paths stop crawlers that harvested links from HTML, because unsigned requests fail at the edge before any bytes move.

## **FAQ: AI bots, AI crawler traffic and robots.txt**

### How much bandwidth do AI bots use on a typical website?

AI bots typically use a few terabytes a month on HTML-only sites and tens of terabytes on sites serving archives or media. Read the Docs reported 73 TB from one AI crawler in May 2024. The deciding variable is bytes per fetch, so measure per token in your CDN logs rather than relying on industry averages.

### Does blocking Google-Extended in robots.txt reduce AI crawler traffic?

No, blocking Google-Extended does not reduce bandwidth, because Googlebot performs the fetches and Google-Extended only controls whether content is used for Gemini. The same applies to Applebot-Extended. To cut bytes, target tokens that fetch on their own, such as GPTBot, ClaudeBot, CCBot or Meta-ExternalAgent, and verify the drop in CDN logs.

### Do AI crawlers respect robots.txt rules?

Major training crawlers such as GPTBot, ClaudeBot and CCBot document robots.txt compliance, but compliance is voluntary. Some user-triggered fetchers are documented as not applying robots.txt rules, and spoofed user agents ignore it entirely. Pair robots.txt with IP verification against published ranges and an edge rate limit that returns 429 to tagged AI crawler tokens.

### Why does AI crawler traffic have a lower cache hit ratio than human traffic?

AI crawlers request long-tail URLs that humans rarely visit, so those objects are usually cold in cache. Query parameter permutations fragment the cache key further. A miss ratio of 30% to 60% for crawler traffic is a reasonable estimate, against roughly 5% for humans. Negative caching, key normalization and request coalescing close most of that gap.

## **Run the bot egress audit this week**

Pull 30 days of CDN logs and run Steps 1 to 5. Produce one table with bytes, requests, bytes per fetch and hit ratio for each AI crawler token, then attach both the CDN figure and the origin figure. That table tells you whether robots.txt groups, cache key work or rate limits deserve the next sprint, and it is worth having even if you change nothing else. Re-run it a week after the changes to confirm the drop.

If the bytes that remain after these rules are mostly large files, the [**BlazingCDN cost calculator**](https://blazingcdn.com/cdn-cost-calculator/) prices that residual traffic on progressive tiers from $5 per TB down to $2.50 per TB at volume, with no request fees, as of November 2026.

Share: [f](https://www.facebook.com/sharer/sharer.php?u=https://blog.blazingcdn.com/en-us/ai-bots-and-your-bandwidth-bill-measuring-ai-crawler-traffic) [in](https://www.linkedin.com/sharing/share-offsite/?url=https://blog.blazingcdn.com/en-us/ai-bots-and-your-bandwidth-bill-measuring-ai-crawler-traffic) [𝕏](https://twitter.com/intent/tweet?url=https://blog.blazingcdn.com/en-us/ai-bots-and-your-bandwidth-bill-measuring-ai-crawler-traffic&text=) [✉](mailto:?subject=%3Cspan%20id="hs_cos_wrapper_name"%20class="hs_cos_wrapper%20hs_cos_wrapper_meta_field%20hs_cos_wrapper_type_text"%20style=""%20data-hs-cos-general-type="meta_field"%20data-hs-cos-type="text"%20%3EAI%20Bots%20and%20Your%20Bandwidth%20Bill:%20Measuring%20AI%20Crawler%20Traffic%3C/span%3E&body=https://blog.blazingcdn.com/en-us/ai-bots-and-your-bandwidth-bill-measuring-ai-crawler-traffic)

![BlazingCDN](https://blog.blazingcdn.com/hs-fs/hubfs/Logo/blog-logo-w.png?height=24&name=blog-logo-w.png)

*Heavy traffic.*  
Light bill.

The CDN for video and large traffic

100 TB a month at list price

- BlazingCDN**$415**
- [bunny.net](https://bunny.net/pricing/) Volume**$500**
- [CDN77](https://www.cdn77.com/pricing), plan to 250 TB**$990**
- [KeyCDN](https://www.keycdn.com/pricing), NA and Europe**$2,600**
- [CloudFront](https://aws.amazon.com/cloudfront/pricing/pay-as-you-go/) pay as you go**$6,965**

Each provider's published list price for transfer, lowest first, checked November 2026. BlazingCDN publishes this blog. Up to 50 TB a month a flat plan such as CloudFront Pro ($15) can cost less.

[Calculate your cost](https://blazingcdn.com/cdn-cost-calculator/?utm_source=blog&utm_medium=sidebar&utm_campaign=blog_sidebar&utm_content=compare_calc)

## Related posts

[![Egress vs Ingress: Network Traffic and Cloud Costs](https://blog.blazingcdn.com/hubfs/Gemini-Blog/image-Nov-03-2026-05-15-39-4044-PM.jpeg)](https://blog.blazingcdn.com/en-us/egress-vs-ingress-network-traffic-and-cloud-costs?hsLang=en-us)

Infrastructure

### [Egress vs Ingress: Network Traffic and Cloud Costs](https://blog.blazingcdn.com/en-us/egress-vs-ingress-network-traffic-and-cloud-costs?hsLang=en-us)

Egress vs ingress comes down to direction and who pays. Ingress is traffic entering a network or cloud account, and ...

Nov 3, 2026, 6:18:43 PM [Read more](https://blog.blazingcdn.com/en-us/egress-vs-ingress-network-traffic-and-cloud-costs?hsLang=en-us)

[![Cloud Cost Management Tools: 75 CTO-Tested Picks for 2026](https://blog.blazingcdn.com/hubfs/Gemini-Blog/image-May-13-2026-11-32-36-1432-AM.png)](https://blog.blazingcdn.com/en-us/75-top-startup-ctos-cost-cutting-tools-2025?hsLang=en-us)

Infrastructure

### [Cloud Cost Management Tools: 75 CTO-Tested Picks for 2026](https://blog.blazingcdn.com/en-us/75-top-startup-ctos-cost-cutting-tools-2025?hsLang=en-us)

Cloud Cost Management Tools: 75 CTO-Tested Picks for 2026 The median Series A startup now burns $47,000 per month on ...

Oct 16, 2026, 9:30:23 AM [Read more](https://blog.blazingcdn.com/en-us/75-top-startup-ctos-cost-cutting-tools-2025?hsLang=en-us)

[![AWS cost optimization](https://blog.blazingcdn.com/hubfs/services_1_optimizaiton.png)](https://blog.blazingcdn.com/en-us/aws-cost-optimization-guide-2024?hsLang=en-us)

Infrastructure

### [AWS Cost Optimization: A Practical Guide](https://blog.blazingcdn.com/en-us/aws-cost-optimization-guide-2024?hsLang=en-us)

Amazon Web Services (AWS) stands out as a pivotal platform for organizations seeking to leverage the power of the ...

Aug 6, 2026, 11:10:23 PM [Read more](https://blog.blazingcdn.com/en-us/aws-cost-optimization-guide-2024?hsLang=en-us)

[![Reduce Your AWS Bill By 50%](https://cdn.leonardo.ai/users/50b20153-23ad-4a4e-9850-7b347a32af23/generations/36a75ac8-7de1-4ad8-8d1b-d234ba891edf/Default_Reduce_Your_AWS_Bill_By_50_0.jpg)](https://blog.blazingcdn.com/en-us/reduce-your-aws-bill-by-50-percent?hsLang=en-us)

Infrastructure

### [Reduce Your AWS Bill By 50%](https://blog.blazingcdn.com/en-us/reduce-your-aws-bill-by-50-percent?hsLang=en-us)

You’ve likely heard the horror stories. That nauseating moment your AWS bill arrives, and it’s shockingly higher than ...

Aug 19, 2024, 3:31:30 PM [Read more](https://blog.blazingcdn.com/en-us/reduce-your-aws-bill-by-50-percent?hsLang=en-us)

[![BlazingCDN](https://blog.blazingcdn.com/hubfs/Logo/blog-logo-w.png)](https://blog.blazingcdn.com?hsLang=en-us)

[📘 Learn](https://blog.blazingcdn.com/cdn-learn?hsLang=en-us) [📊 Benchmarks](https://blog.blazingcdn.com/cdn-ratings-and-benchmarks?hsLang=en-us) [🎬 Video & Streaming](https://blog.blazingcdn.com/video-streaming-cdn?hsLang=en-us) [🏭 Industries](https://blog.blazingcdn.com/cdn-industry-insights?hsLang=en-us) [💰 Pricing & Costs](https://blog.blazingcdn.com/cdn-pricing-and-cdn-costs?hsLang=en-us) [⚡ Compare](https://blog.blazingcdn.com/cdn-comparison?hsLang=en-us) [🔒 Security](https://blog.blazingcdn.com/cdn-security?hsLang=en-us) [🛠️ Tools](https://blog.blazingcdn.com/cdn-tools?hsLang=en-us) [✍️ Publish with us](https://blog.blazingcdn.com/publish-with-us?hsLang=en-us)

in f 𝕏 ✉

 Copyright © BlazingCDN |. All rights reserved.

```json
{
  "@context" : "https://schema.org",
  "@id" : "https://blazingcdn.com/#organization",
  "@type" : "Organization",
  "description" : "High-volume CDN for video, live streaming, OTT/IPTV, software, games, SaaS and large-file delivery.",
  "logo" : {
    "@id" : "https://blazingcdn.com/#logo",
    "@type" : "ImageObject",
    "height" : 560,
    "url" : "https://blazingcdn.com/wp-content/uploads/2024/08/logo-560-560.png",
    "width" : 560
  },
  "name" : "BlazingCDN",
  "sameAs" : [ "https://www.linkedin.com/company/blazingcdn", "https://x.com/BlazingCdn", "https://www.reddit.com/r/BlazingCDN/", "https://www.wikidata.org/wiki/Q141711074" ],
  "url" : "https://blazingcdn.com/"
}
```

```json
{
  "@context" : "https://schema.org",
  "@id" : "https://blog.blazingcdn.com/#website",
  "@type" : "WebSite",
  "inLanguage" : "en-US",
  "name" : "BlazingCDN Blog",
  "publisher" : {
    "@id" : "https://blazingcdn.com/#organization"
  },
  "url" : "https://blog.blazingcdn.com/"
}
```

```json
{
  "@context" : "https://schema.org",
  "@id" : "https://blog.blazingcdn.com/en-us/ai-bots-and-your-bandwidth-bill-measuring-ai-crawler-traffic#article",
  "@type" : "BlogPosting",
  "author" : {
    "@id" : "https://blog.blazingcdn.com/en-us/author/blazingcdn#person",
    "@type" : "Person",
    "description" : "Vadym Borovskyi is Chief Product and Marketing Officer at BlazingCDN, where he has led product strategy and growth since 2021. He has more than 11 years of experience in IT product management and B2B marketing. On this blog he writes about CDN pricing and egress costs, video and software delivery, cache configuration, and how teams evaluate and switch CDN providers. He is based in Warsaw, Poland.",
    "image" : "https://143144902.fs1.hubspotusercontent-eu1.net/hubfs/143144902/vadym-borovskyi.jpg",
    "jobTitle" : "Chief Product and Marketing Officer",
    "knowsAbout" : [ "Content Delivery Networks (CDN)", "CDN pricing", "Video streaming delivery", "Software and game distribution", "Cache optimization", "Product Management", "B2B Marketing", "Digital Marketing", "Pay Per Click Marketing (PPC)", "Google Ads", "Search Engine Marketing (SEM)", "Marketing Strategy", "Artificial Intelligence (AI)", "Cloud Management", "SaaS Development", "IT Consulting", "Computer Networking", "Project Management", "Startups" ],
    "name" : "Vadym Borovskyi",
    "sameAs" : [ "https://www.linkedin.com/in/axelt/" ],
    "url" : "https://blog.blazingcdn.com/en-us/author/blazingcdn",
    "worksFor" : {
      "@id" : "https://blazingcdn.com/#organization"
    }
  },
  "dateModified" : "2026-11-19T15:11:04Z",
  "datePublished" : "2026-11-19T15:11:04Z",
  "description" : "Measure AI crawler traffic, see how AI bots consume bandwidth, and use robots.txt AI rules to control crawler access and costs.",
  "headline" : "AI Bots and Your Bandwidth Bill: Measuring AI Crawler Traffic",
  "image" : "https://143144902.fs1.hubspotusercontent-eu1.net/hubfs/143144902/Gemini-Blog/image-Nov-19-2026-03-08-22-6265-PM.jpeg",
  "inLanguage" : "en-us",
  "isPartOf" : {
    "@id" : "https://blog.blazingcdn.com/#website"
  },
  "mainEntityOfPage" : {
    "@id" : "https://blog.blazingcdn.com/en-us/ai-bots-and-your-bandwidth-bill-measuring-ai-crawler-traffic",
    "@type" : "WebPage"
  },
  "publisher" : {
    "@id" : "https://blazingcdn.com/#organization"
  },
  "wordCount" : 2225
}
```

```json
{
  "@context" : "https://schema.org",
  "@id" : "https://blog.blazingcdn.com/en-us/ai-bots-and-your-bandwidth-bill-measuring-ai-crawler-traffic#breadcrumb",
  "@type" : "BreadcrumbList",
  "itemListElement" : [ {
    "@type" : "ListItem",
    "item" : "https://blog.blazingcdn.com/en-us",
    "name" : "Blog",
    "position" : 1
  }, {
    "@type" : "ListItem",
    "item" : "https://blog.blazingcdn.com/en-us/ai-bots-and-your-bandwidth-bill-measuring-ai-crawler-traffic",
    "name" : "AI Bots and Your Bandwidth Bill: Measuring AI Crawler Traffic",
    "position" : 2
  } ]
}
```

```json
{
  "@context" : "https://schema.org",
  "@id" : "https://blog.blazingcdn.com/en-us/ai-bots-and-your-bandwidth-bill-measuring-ai-crawler-traffic#faq",
  "@type" : "FAQPage",
  "mainEntity" : [ {
    "@type" : "Question",
    "acceptedAnswer" : {
      "@type" : "Answer",
      "text" : "AI bots typically use a few terabytes a month on HTML-only sites and tens of terabytes on sites serving archives or media. Read the Docs reported 73 TB from one AI crawler in May 2024. The deciding variable is bytes per fetch, so measure per token in your CDN logs rather than relying on industry averages."
    },
    "name" : "How much bandwidth do AI bots use on a typical website?"
  }, {
    "@type" : "Question",
    "acceptedAnswer" : {
      "@type" : "Answer",
      "text" : "No, blocking Google-Extended does not reduce bandwidth, because Googlebot performs the fetches and Google-Extended only controls whether content is used for Gemini. The same applies to Applebot-Extended. To cut bytes, target tokens that fetch on their own, such as GPTBot, ClaudeBot, CCBot or Meta-ExternalAgent, and verify the drop in CDN logs."
    },
    "name" : "Does blocking Google-Extended in robots.txt reduce AI crawler traffic?"
  }, {
    "@type" : "Question",
    "acceptedAnswer" : {
      "@type" : "Answer",
      "text" : "Major training crawlers such as GPTBot, ClaudeBot and CCBot document robots.txt compliance, but compliance is voluntary. Some user-triggered fetchers are documented as not applying robots.txt rules, and spoofed user agents ignore it entirely. Pair robots.txt with IP verification against published ranges and an edge rate limit that returns 429 to tagged AI crawler tokens."
    },
    "name" : "Do AI crawlers respect robots.txt rules?"
  }, {
    "@type" : "Question",
    "acceptedAnswer" : {
      "@type" : "Answer",
      "text" : "AI crawlers request long-tail URLs that humans rarely visit, so those objects are usually cold in cache. Query parameter permutations fragment the cache key further. A miss ratio of 30% to 60% for crawler traffic is a reasonable estimate, against roughly 5% for humans. Negative caching, key normalization and request coalescing close most of that gap."
    },
    "name" : "Why does AI crawler traffic have a lower cache hit ratio than human traffic?"
  } ]
}
```