OTA Updates Explained: Delivering Firmware at Scale

<span id="hs_cos_wrapper_name" class="hs_cos_wrapper hs_cos_wrapper_meta_field hs_cos_wrapper_type_text" style="" data-hs-cos-general-type="meta_field" data-hs-cos-type="text" >OTA Updates Explained: Delivering Firmware at Scale</span>

OTA updates move a signed software package from a vendor's firmware update server to a device in the field, verify it cryptographically, install it into an inactive partition, and commit only after the new image boots cleanly. In a well-run pipeline, a delta package is typically 5 to 25 percent of the full image (an estimate that depends on how much of the binary changed). The rollout expands in stages starting at about 1 percent of the fleet. A CDN serves the heavy bytes, and the origin only decides which device gets which release.

In cars, regulation has made this formal. Under UNECE Regulation No. 156, every new vehicle registered in the European Union since July 2024 must be covered by a certified software update management system, so automotive OTA pipelines are now audited release engineering, not a download link.

How an OTA firmware update travels from build to boot

The pipeline has eight stages, and each one fails in its own way. Fleets that ship weekly run each stage as a separate service with its own metrics.

  1. Build and diff: CI produces the full image plus a delta from each supported source version.
  2. Sign: an offline or HSM-backed key signs the image hash and the metadata that describes it.
  3. Publish: blobs go to object storage under content-addressed names. Metadata goes to the update service.
  4. Check-in: the device reports hardware ID, current version and cohort. The server answers with a small manifest or with "nothing for you".
  5. Download: the device fetches the blob from a CDN hostname and resumes with HTTP range requests after a dropped connection.
  6. Verify: the device hashes the stream during download and checks the signature against a key pinned in the device.
  7. Install: the image is written to the inactive slot, or applied as a snapshot on Android Virtual A/B. It never overwrites the running system.
  8. Commit or revert: after reboot, the new image must pass a health check and mark itself good. Otherwise the bootloader falls back to the previous slot.

What is inside an over the air update package

An update has two parts that should never be confused. The payload is large, immutable and identical for every device. The metadata is small, signed, short-lived and specific to the device asking for it.

Android's update_engine consumes a payload.bin made of install operations plus a signed payload properties file. Vehicle stacks that follow Uptane split metadata across two repositories. A Director repository issues per-vehicle instructions, and an Image repository signs the set of images that exist, so compromising one server is not enough to push malicious firmware.

Full images vs delta packages

Delta generators trade server-side CPU and storage for fewer bytes on the wire. Common examples are bsdiff, Google's Zucchini (used by Chromium), and puffin, which lets Android diff inside deflate streams. The catch is combinatorial: a delta is valid only against one exact source build. Supporting the last five versions therefore means five deltas per release, plus a full image as the fallback for everything older.

Chunk-based schemes avoid per-version deltas. RAUC's adaptive updates and casync-style chunking let the device fetch only the blocks whose hashes it lacks. These chunks also cache better on a CDN, because consecutive releases share most of them.

How integrity checks stop bad or malicious firmware

A hash proves the bytes match the manifest, and a signature proves who wrote the manifest. Neither one covers the attacks The Update Framework (TUF) was designed for:

  • Rollback: serving an older build that is still validly signed but vulnerable.
  • Freeze: hiding a newer release from the device.
  • Mix-and-match: combining metadata from different points in time.

The defenses for these attacks are:

  • Expiring metadata: TUF's timestamp role signs a file valid for hours or a day, so a device notices when it is being shown stale data.
  • Anti-rollback counters: Android Verified Boot keeps a rollback index in tamper-evident storage and refuses any image with a lower index.
  • Threshold signatures: root and targets roles can require k of n keys, so one stolen key cannot sign a release.
  • Layered verification: the device verifies the stream before writing the boot slot, then the secure boot chain verifies the image again at every boot.

The detail that bites in production is time. Expiring metadata assumes the device knows the date, but many IoT boards without a battery-backed real-time clock boot believing it is 1970 until NTP answers. Either order the boot sequence so time sync completes before metadata checks, or define an explicit, bounded skew window.

Staged rollouts for OTA updates at fleet scale

A staged rollout releases to a deterministic cohort and expands only when health gates pass. The cohort is usually a hash of the device ID modulo 100 or 10,000. Typical ladders run 1, 5, 25, 50 and 100 percent. Each stage soaks long enough to cover a full daily usage cycle: 24 to 72 hours for phones and IoT devices, and longer for vehicles that may sit parked for days.

Gate on what devices report after commit:

  • install success rate;
  • slot fallbacks after reboot;
  • crash rate per hour of runtime;
  • battery or thermal regressions.

Halting a rollout is a control-plane flag that takes effect on each device's next check-in. Your check-in interval is therefore your worst-case reaction time to a bad release.

Worked calculation: what a 25 percent stage does to egress

The scenario below uses this blog's assumptions, not measurements:

  • a fleet of 1 million devices;
  • a 1.2 GB full image and a 250 MB delta;
  • a 25 percent stage (250,000 devices) that completes within 48 hours.

The "no jitter" rows assume every device checks in at the same nightly maintenance time, so 60 percent of the stage downloads in the first 2 hours.

Scenario Bytes delivered Window Average egress
Delta, check-ins jittered across 48 h 62.5 TB 48 h 2.9 Gb/s
Full image, check-ins jittered across 48 h 300 TB 48 h 13.9 Gb/s
Delta, no jitter, 60 percent in first 2 h 37.5 TB 2 h 41.7 Gb/s
Full image, no jitter, 60 percent in first 2 h 180 TB 2 h 200 Gb/s

Deltas cut peak egress by a factor of 4.8 (13.9 to 2.9 Gb/s), while jittering check-ins cuts it by about 14 (41.7 to 2.9 Gb/s), so the polling schedule moves the peak more than the package format does.

The fix is almost free. Add several hours of random jitter to the check-in schedule. When a stage is saturated, have the server answer with a Retry-After header carrying a randomized delay.

Firmware update server vs CDN: who serves what

The firmware update server should never stream payloads. Its job is to answer small, authenticated, uncacheable check-ins. The blobs live behind a CDN on immutable, content-addressed URLs, with the file's SHA-256 in the path. Serve them with Cache-Control set to public, a max-age of one year and the immutable directive. Every release gets a new URL, so there is never anything to purge.

If you run the cache tier yourself on nginx, four directives do most of the work:

  • slice set to 1 MB or larger, so a range request for the middle of a 1.2 GB image does not force a full-object fill. Include the slice_range variable in proxy_cache_key and pass it upstream in the Range header.
  • proxy_cache_valid for 200 and 206 responses, set to 30 days or more, since content-addressed objects never change.
  • proxy_cache_lock set to on, with proxy_cache_lock_timeout around 10 seconds, so thousands of simultaneous misses collapse into one upstream fetch.
  • proxy_cache_use_stale with the updating parameter, so a slow origin does not stall a rollout.

Two more rules prevent cache fragmentation:

  • If URLs are signed for authorization, exclude the token and expiry parameters from the cache key.
  • Pre-warm the shield and the busiest edges with new objects before raising a stage, not after.

Software update delivery is one of the workloads BlazingCDN is built for. Its Files on Demand delivery for firmware and installers serves objects up to 50 GB, with origin shield, request coalescing, signed URLs and cache warm-up through the API on every plan. On its published tiers as of November 2026, with no request fees, the 1,200 TB full-image release to the 1 million devices above prices at $3,815, and the 250 TB delta release at $940.

Trade-offs, failure modes and observability gaps

  • A/B costs flash. Duplicate system partitions double storage. Android Virtual A/B shrinks this with compressed copy-on-write snapshots, but adds a merge phase after reboot during which another update cannot be applied.
  • Deltas multiply artifacts. Devices that skipped several releases need a full image anyway, so you always pay for the full-image path.
  • Microcontrollers need an explicit confirm. On ESP-IDF with app rollback enabled, a new image boots in a pending-verify state. It must call esp_ota_mark_app_valid_cancel_rollback, or the next reset reverts it. MCUboot's test-then-confirm swap works the same way.
  • Metered links change policy. Cellular modules and vehicles should defer large downloads to Wi-Fi or to parked periods when the policy allows it.
  • CDN logs show bytes, not installs. A completed 206 response says nothing about whether the device verified the image and committed. Join CDN logs by object hash with device telemetry, or a 99 percent download success rate will hide a 3 percent fallback rate.

Verify this on your own stack

  • Request a 1 MB range from the middle of a freshly published image twice. The second response should show a cache HIT and a growing Age header, and origin logs should show one sliced fetch, not a full-object pull.
  • Graph origin requests per object hash during a stage expansion. With shield and coalescing working, the count stays in single digits per release.
  • Plot device check-in times as a histogram. Any spike above 3 times the mean is your thundering herd.

Who this architecture fits, and who it does not

The split between a control plane and a CDN data plane pays off for fleets above roughly 10,000 devices, or for payloads above 100 MB. At that scale, a single release reaches tens of terabytes, and origin egress plus stage-expansion peaks dominate the cost.

It fits poorly in two cases:

  • Constrained NB-IoT sensors. These receive packages of tens of kilobytes through the LwM2M Firmware Update object (Object 5), using block-wise CoAP transfer, where a CDN adds little.
  • Air-gapped industrial sites. Updates there arrive on signed media or through a local mirror.

FAQ: OTA updates for phones, cars and IoT devices

How do OTA updates avoid bricking a device during installation?

They write the new image to an inactive partition and switch only after it boots successfully. A/B layouts on Android, RAUC and Mender keep the old slot intact, so a power loss mid-write leaves the device on its current version. After reboot, the new image must mark itself healthy, or the bootloader reverts to the previous slot automatically.

How much smaller is a delta OTA firmware update than a full image?

A delta is typically 5 to 25 percent of the full image, an estimate that depends on how much of the binary changed between builds. Compressed images diff poorly unless the tool works inside the compression, which is why Android uses puffin for deflate streams. Every delta is valid against only one exact source version.

Should a firmware update server deliver the firmware files itself?

No. The firmware update server should answer check-ins and hand out signed manifests, while a CDN serves the payloads. Check-ins are small, per-device and uncacheable, whereas firmware blobs are large and identical across the fleet. Content-addressed URLs with long cache lifetimes, origin shield and request coalescing keep origin load near one fetch per object per release.

How quickly should an over the air update rollout expand?

Expand only after each stage survives a full usage cycle with healthy metrics, typically 24 to 72 hours per stage for phones and IoT devices. Common ladders run 1, 5, 25, 50 and 100 percent of the fleet. Vehicles need longer soaks because many sit parked for days, and how fast you can halt depends on how often devices check in.

Model your next rollout before the first cohort ships

This week, pull three numbers from your last release:

  • payload size per device;
  • fleet size per stage;
  • the check-in time histogram from your update server logs.

Rebuild the egress table above with your figures, and compare the jittered peak with the unjittered one. If the gap exceeds 5 times, fix the polling schedule before touching delta tooling. Then run the mid-object range test against your current CDN to confirm that slicing and coalescing actually protect the origin.

If your fleet pulls firmware or installers at tens of terabytes per release, the BlazingCDN software delivery page describes how its 14-day testing period on real production traffic works, at $5 per TB with the monthly minimum waived.

Heavy traffic.
Light bill.

The CDN for video and large traffic