CDN Log Delivery: Real-Time Logs and Analytics Pipelines

A working cdn log delivery pipeline takes about 40 minutes to stand up on an existing object storage bucket plus one collector host. The end state: edge access logs landing in hourly partitions, a normalized cache-status field, and a cache hit ratio that reconciles with your CDN's own dashboard within 2%. Two delivery modes exist and they are not interchangeable. Real-time streaming puts lines in front of a query engine in 5–60 seconds; batch push writes compressed files every 5–15 minutes, sometimes hourly. Most teams need both, routed into one store.

How cdn log delivery works: real-time streaming vs batch push

Every edge node writes a line per request into a local buffer. From there the CDN either ships the buffer over a persistent connection to a customer endpoint or message bus (streaming), or rolls it into a compressed object and writes it to your bucket on a time or size trigger (batch push). Both are at-least-once. Neither is ordered.

The field set is usually configurable, and that choice is permanent for historical data. Fields you do not enable today cannot be backfilled tomorrow.

Use streaming for decisions measured in seconds and batch push for anything you would happily re-run a week later.

Prerequisites

• A CDN account with log delivery enabled and permission to edit the field set.
• An object storage bucket plus a write-scoped credential for the CDN or your collector.
• One collector host, 4 vCPU and 8 GB RAM, with 50 GB of local disk for buffering.
• A query engine that reads columnar files from object storage.
• Clocks in UTC everywhere. Mixed local time zones are the single most common cause of phantom traffic gaps.

Build the cdn analytics pipeline step by step

Step 1: Pick the cdn log delivery mode and size it

Enable batch push first. It is the cheaper baseline and it is the source of truth for reconciliation. Add streaming only for the subset of traffic where a 10-minute delay changes an action.

A delivery tier averaging 1,000 requests per second produces roughly 3.6 million log lines per hour. At about 450 bytes per line in an extended W3C-style access format, that is near 1.6 GB per hour uncompressed, or about 160 MB per hour with zstd compression at level 3 in 2026 measurements. Over 30 days: roughly 1.15 TB raw, 115 GB stored.

Step 2: Pin the field schema before you ship anything

Write the schema down as a file in your repo. Rename vendor fields to your own names at ingest, so a future CDN swap changes one transform instead of every dashboard.

schema_version: 1
fields:
  - timestamp_ms        # event time at the edge, UTC epoch millis
  - request_id          # vendor request identifier, used for dedupe
  - client_ip           # from the CDN field, never the socket peer
  - host
  - path                # query string stripped at ingest
  - method
  - status
  - bytes_sent
  - cache_status        # raw vendor value, normalized downstream
  - ttfb_ms             # edge time to first byte
  - origin_time_ms      # present only on misses
  - protocol            # h2, h3, http1.1
  - tls_version
  - user_agent
  - country
  - pop_region          # coarse geography bucket, not node identity

Step 3: Normalize cache status and tenant at ingest

Cache status vocabularies differ across vendors: hit, HIT, cached, hit-stale, revalidated. Collapse them into a boolean plus a reason code once, at ingest. Do not push that logic into dashboards, where it will drift.

sources:
  cdn_stream:
    type: http_server
    address: 0.0.0.0:8080
    decoding:
      codec: json

transforms:
  normalize:
    type: remap
    inputs: [cdn_stream]
    source: |
      .cache = downcase(string!(.cache_status))
      .is_hit = includes(["hit", "cached", "hit_refresh", "revalidated"], .cache)
      .tenant = split(string!(.host), ".")[0]
      .path = split(string!(.path), "?")[0]
      del(.user_agent_raw)

sinks:
  lake:
    type: aws_s3
    inputs: [normalize]
    bucket: LOGS_BUCKET_NAME        # substitute your bucket name
    key_prefix: "cdn/dt=%Y-%m-%d/hr=%H/"
    compression: zstd
    encoding:
      codec: json
    batch:
      max_bytes: 268435456
      timeout_secs: 300
    buffer:
      type: disk
      max_size: 21474836480

Step 4: Partition on event time, compact, set retention

Partition by event time, not arrival time. Late files are normal in batch push, and arrival-time partitioning silently smears an outage across two hours. Compact small objects into 256 MB columnar files once per hour; query cost on a well-compacted hour is typically 5–10x lower than on raw JSON.

pipeline:
  partition_keys: [dt, hr, tenant]
  partition_time_source: event_time
  compaction:
    format: parquet
    target_file_bytes: 268435456
    sort_keys: [tenant, timestamp_ms]
  retention:
    hot_days: 30
    cold_days: 400
alerts:
  delivery_lag_seconds:
    stream_warn: 90
    stream_page: 600
    batch_warn: 1200
  hourly_line_count_drop_pct: 20
  duplicate_request_id_pct: 1.0

Validation: is the cdn log delivery complete?

1. Count lines in one closed hour partition and compare with the CDN's reported request count for the same UTC hour. Expect agreement within 1–2%.
2. Measure delivery lag as processing time minus event time, p99 per batch. Expect under 90 seconds for streaming, under 20 minutes for batch push.
3. Check field completeness: cache status non-empty on near 100% of lines, origin time present only on misses.
4. Count distinct request identifiers against total lines. A duplicate rate above 1% means your dedupe window is too short.

Rollback

Disable the sink in the CDN console. That single toggle stops the write path; the collector drains its disk buffer within minutes. Apply a one-day expiry lifecycle rule to the test prefix rather than deleting objects by hand, so in-flight writes do not error. Keep the schema file. One honest caveat: while streaming is off, those lines are gone permanently, and most batch push implementations backfill a limited window or nothing at all.

Failure modes: symptom, cause, fix

Silence is the dangerous failure mode: a pipeline receiving nothing reports perfect health unless you alert on delivery lag and hourly line counts.

Tuning real-time cdn logs once the pipeline is stable

Sample asymmetrically. Keep 100% of non-2xx responses and 1–10% of successful static asset hits; on a large software delivery workload this cuts volume by 70–90% while preserving every diagnostic signal. Strip query strings at ingest unless you have a specific attribution need, since unbounded path cardinality dominates columnar storage cost.

Vendor behavior differs in ways worth checking during selection: which fields are exposed, whether both delivery modes exist, and how quickly the schema can be changed. BlazingCDN handles log delivery configuration per account with roughly one-hour onboarding, which matters when your schema decision needs to be made before launch rather than after. Review the available CDN delivery and configuration features against the field list in Step 2 before you commit.

Trade-off worth naming: this pipeline tells you what the edge saw, not what the browser experienced. Edge time to first byte excludes last-mile latency and render time. Pair it with client-side timing data, or you will optimize a number your users never feel.

FAQ: cdn log delivery and analytics pipelines

What is the difference between real-time CDN logs and batch log push?

Real-time CDN logs stream individual request records to an endpoint within 5–60 seconds of the request, while batch push writes compressed files to object storage every 5–15 minutes. Streaming supports live triage and abuse detection. Batch push is cheaper, more durable under consumer failure, and remains the correct source for billing reconciliation.

How much storage does a CDN analytics pipeline need?

Budget roughly 450 bytes per uncompressed log line and a 10:1 compression ratio with zstd. At 1,000 requests per second that is about 1.15 TB raw and 115 GB stored per 30 days. Sampling successful static responses at 10% typically reduces stored volume by 70–90% without losing error visibility.

Can CDN log delivery replace real user monitoring?

No. CDN log delivery records what the edge served, including cache status, status codes, and edge time to first byte, but it cannot observe last-mile latency, connection setup on the client, or rendering. Use logs for delivery-side truth and client instrumentation for experience metrics, then join them on a shared request identifier.

Why do CDN log counts not match the CDN dashboard?

Mismatches usually come from three sources: at-least-once duplicates inflating counts by 0.1–1%, time zone or arrival-time partitioning shifting requests between hours, and dashboards that measure bytes where the pipeline measures requests. Reconcile on a closed UTC hour, dedupe by request identifier, and compare like units.

Run the reconciliation this week

Pick one closed UTC hour from yesterday. Count lines in your partition, count distinct request identifiers, and compare both against the request count your CDN reports for that same hour. If the gap exceeds 2%, you have either a dedupe problem or a partitioning problem, and the distinct-count delta tells you which. Then instrument delivery lag as p99 of processing time minus event time and alert on it. That one metric catches the failure mode that every other alert misses: logs that simply stopped arriving.