CoreOS Container Linux developer SDK guide

These are the instructions for building Container Linux itself. By the end of the guide you will build a developer image that you can run under KVM and have tools for making changes to the code.

Container Linux is an open source project. All of the source for Container Linux is available on github. If you find issues with these docs or the code please send a pull request.

Direct questions and suggestions to the IRC channel or mailing list.

Getting started

Let's get set up with an SDK chroot and build a bootable image of Container Linux. The SDK chroot has a full toolchain and isolates the build process from quirks and differences between host OSes. The SDK must be run on an x86-64 Linux machine, the distro should not matter (Ubuntu, Fedora, etc).

Prerequisites

System requirements to get started:

  • curl
  • git
  • python2

You also need a proper git setup:

git config --global user.email "you@example.com"
git config --global user.name "Your Name"

NOTE: Do the git configuration as a normal user and not with sudo.

Option 1: Using Cork

The cork utility, included in the CoreOS mantle project, can be used to create and work with an SDK chroot.

In order to use this utility, you must additionally have the golang toolchain installed and configured correctly.

First, install the cork utility:

git clone https://github.com/coreos/mantle
cd mantle
./build cork
mkdir ~/bin
mv ./bin/cork ~/bin
export PATH=$PATH:$HOME/bin

You may want to add the PATH export to your shell profile (e.g. .bashrc).

Next, the cork utility can be used to create an SDK chroot:

mkdir coreos-sdk
cd coreos-sdk
cork create
cork enter

Note: The create and enter commands will request root permissions via sudo.

To use the SDK chroot in the future, run cork enter from the above directory.

Option 2: using repo and cros_sdk

Install Repo

The repo utility helps to manage the collection of git repositories that makes up Container Linux.

For newer Debian, Ubuntu, and other Debian based systems, install the repo package from your distro:

sudo apt-get install repo

For systems without a packaged repo download it and add it to $PATH:

mkdir ~/bin
export PATH="$PATH:$HOME/bin"
curl https://storage.googleapis.com/git-repo-downloads/repo > ~/bin/repo
chmod a+x ~/bin/repo

You may want to add the PATH export to .bashrc or /etc/profile.d/ so that you don’t need to set $PATH in every new shell.

Bootstrap the SDK chroot

Create a project directory. This will hold all of your git repos and the SDK chroot. A few gigabytes of space will be necessary.

mkdir coreos; cd coreos

Initialize the .repo directory with the manifest that describes all of the git repos required to get started.

repo init -u https://github.com/coreos/manifest.git

Synchronize all of the required git repos from the manifest.

repo sync

Use cros_sdk to setup the chroot

Download and enter the SDK chroot which contains all of the compilers and tooling.

./chromite/bin/cros_sdk

WARNING: To delete the SDK chroot, use ./chromite/bin/cros_sdk --delete. Otherwise, you will delete /dev entries that are bind-mounted into the chroot.

Using QEMU for cross-compiling

The Container Linux initramfs is generated with the dracut tool. Dracut assumes it is running on the target system, and produces output only for that CPU architecture. In order to create initramfs files for other architectures, dracut is executed under QEMU's user mode emulation of the target CPU via the host system's binfmt support.

Configuring QEMU for 64 bit ARM binaries

Note that "64 bit ARM" is known by two short forms: aarch64 (as seen in the configuration file for QEMU), and arm64 (as seen in how Container Linux and many other distributions refer to the architecture).

The QEMU binary, /usr/bin/qemu-aarch64-static is not expected to be on the host workstation. It will be inside the arm64-usr build chroot entered before running dracut.

Configuring Debian based systems

For Debian, Ubuntu, and other Debian based systems installing the following packages will configure the host system such that QEMU will be the runtime for 64 bit ARM binaries:

sudo apt-get install binfmt-support qemu-user-static
Configuring other systemd based systems

The cork utility can setup QEMU binfmt support on systemd based systems:

cork setup
systemctl restart systemd-binfmt.service

This will install the configuration file /etc/binfmt.d/qemu-aarch64.conf, which controls how binaries for a given architecture are handled.

Building an image

After entering the chroot via cork or cros_sdk for the first time, you should set user core's password:

./set_shared_user_password.sh

This is the password you will use to log into the console of images built and launched with the SDK.

Selecting the architecture to build

64 bit AMD: The amd64-usr target

The --board option can be set to one of a few known target architectures, or system "boards", to build for a given CPU.

To create a root filesystem for the amd64-usr target beneath the directory /build/amd64-usr/:

./setup_board --default --board=amd64-usr
64 bit ARM: The arm64-usr target

Similarly, use arm64-usr for the cross-compiled ARM target. If switching between different targets in a single SDK, you can add the --board= option to the subsequent build_packages, build_image, and other similar commands to select the given target architecture and path.

./setup_board --default --board=arm64-usr

Build all of the target binary packages:

./build_packages

Render the CoreOS Container Linux image

Build an image based on the binary packages built above, including development tools:

./build_image dev

After build_image completes, it prints commands for converting the raw bin into a bootable virtual machine. Run the image_to_vm.sh command.

Booting

Once you build an image you can launch it with KVM (instructions will print out after image_to_vm.sh runs).

Making changes

git and repo

Container Linux is managed by repo, a tool built for the Android project that makes managing a large number of git repositories easier. From the repo announcement blog:

The repo tool uses an XML-based manifest file describing where the upstream repositories are, and how to merge them into a single working checkout. repo will recurse across all the git subtrees and handle uploads, pulls, and other needed items. repo has built-in knowledge of topic branches and makes working with them an essential part of the workflow.

(from the Google Open Source Blog)

You can find the full manual for repo by visiting android.com - Developing.

Updating repo manifests

The repo manifest for Container Linux lives in a git repository in .repo/manifests. If you need to update the manifest edit default.xml in this directory.

repo uses a branch called 'default' to track the upstream branch you specify in repo init, this defaults to 'origin/master'. Keep this in mind when making changes, the origin git repository should not have a 'default' branch.

Building images

There are separate workflows for building production images and development images.

Tips and tricks

We've compiled a list of tips and tricks that can make working with the SDK a bit easier.

Testing images

Mantle is a collection of utilities used in testing and launching SDK images.