FastPatch

CoreOS is reliably updated via a continous stream of updates

Our Update Philosophy

We believe that frequent, reliable updates are critical to good security. Unfortunately, existing configuration management tools often encounter inconsistent state from machine to machine within a large cluster. This makes running an update on large production deployments even more complex.

CoreOS minimizes the complexity of an update by compartmentalizing each entity that normally needs to be updated: the operating system, application code, and simple configuration values.

Operating System

CoreOS updates are consistent through our use of FastPatch, an active-passive root partition scheme. We update the entire OS as a single unit, instead of package by package.

All CoreOS machines have access to the public Update Service to receive timely updates as soon as they are available. The CoreOS Engineering team manages this service, including the promotion of versions from channel to channel as well as how quickly each update is released. These updates will always be available to all users, regardless of their status as CoreOS customers.

Your engineering team can select specific release channels to deploy and control how your clusters apply updates with update strategies.

Managed Linux customers have access to an additional tool, CoreUpdate, a hosted dashboard that allows for full control over access and downloading of updates. CoreUpdate allows you to create custom channels in addition to the default channels and configure your own roll-out rates.

Read more about CoreUpdate
CoreUpdate screenshot showing a rolling update in progress.

Application Code

By deploying and running your code in containers, each application is packaged with all of its dependencies. This eliminates dependency conflicts and extensive test cases. Containers can be shipped from dev → test → production and will be bit for bit identical in either environment.

Container isolation means you can update each application independently of each other with ease. Even the most poorly coded application can't harm another running on the same machine.

Read more about docker + CoreOS

Configuration Values

Distributed platforms contain hundreds of configuration values. Every cache setting, database address, firewall rule and rate-limit needs to be stored somewhere. Traditionally you update these values via Chef or Puppet. But, you can't audit the state of hundreds of machines before you execute your single config change. What if you triggered a library upgrade on a machine that was missed on the last run?

To solve this problem, CoreOS provides distributed configuration with etcd. A single config value can be changed atomically, and only applications that are listening for that change will be affected. Each application can decide how to react to a value changing, and that logic can be updated independently of other applications.

Read more about etcd + CoreOS

How Operating System Updates Work

CoreOS updates employ a dual-partion scheme that operates differently than most Linux distributions. Instead of updating a single package at a time, CoreOS downloads an entirely new root filesystem and installs it to the passive partition. After the next reboot, CoreOS will be running the latest version.

The dual-partition scheme has many benefits over an in-place upgrade:

  • Safe Roll Back
    The previous, known-to-be-stable version of the operating system is still on the first partition. CoreOS has a built-in failsafe to revert to this partition if an upgrade results in an unstable state. This process can also be executed manually.
  • Signed and Verified
    Each boot partition is read-only, which makes it easy to verify that each download is complete and hasn't been modified in transit.
  • Extremely Fast Execution
    CoreOS boots extremely fast due to its small size. Executing an update requires a reboot which takes just a few seconds, meaning less interruption for your applications. Platforms that support kexec can skip the bootloader process, decreasing reboot time further.

Our update system is based on Google's open-source project, Omaha, that powers updates for the Chrome browser, Chrome OS, Google Earth and more. It features advanced population control and grouping functionality that takes the guesswork out of a rolling upgrade throughout the fleet.

Technical Details

A B Data Update
CoreOS is currently booted off partition A. An update is installed onto partition B.
A B Data
Machine is rebooted onto the B partition.

Initially, your system is booted into the root A partition and CoreOS begins talking to the update service to find out about new updates. If there is an update available it is downloaded and installed to root B. To ensure we don’t disrupt your application, we rate limit the disk and network I/O this process is allowed to use with Linux cgroups.

Using this dual-root scheme is an improvement on the existing workflow of yum or apt-get. Using these tools during upgrading has been known to cause the package manager to force daemons to use new libraries or move configuration files around. With CoreOS, a system update is an atomic operation that can be rolled back.

On CoreOS, the root partition you are running against isn’t modified and your server is never in an unstable or partially upgraded state. To finish off the upgrade, reboot the machine and in a few seconds you will start running on root B with a freshly updated system.