Reading the system log

journalctl is your interface into a single machine's journal/logging and fleetctl journal will fetch the journal for containers started with fleet. All service files and Docker containers insert data into the systemd journal. There are a few helpful commands to read the journal:

Read the entire journal

$ journalctl

-- Logs begin at Fri 2013-12-13 23:43:32 UTC, end at Sun 2013-12-22 12:28:45 UTC. --
Dec 22 00:10:21 localhost systemd-journal[33]: Runtime journal is using 184.0K (max 49.9M, leaving 74.8M of free 499.0M, current limit 49.9M).
Dec 22 00:10:21 localhost systemd-journal[33]: Runtime journal is using 188.0K (max 49.9M, leaving 74.8M of free 499.0M, current limit 49.9M).
Dec 22 00:10:21 localhost kernel: Initializing cgroup subsys cpuset
Dec 22 00:10:21 localhost kernel: Initializing cgroup subsys cpu
Dec 22 00:10:21 localhost kernel: Initializing cgroup subsys cpuacct
Dec 22 00:10:21 localhost kernel: Linux version 3.11.7+ (buildbot@10.10.10.10) (gcc version 4.6.3 (Gentoo Hardened 4.6.3 p1.13, pie-0.5.2)
...
1000s more lines

Read entries for a specific service

Read entries generated by a specific unit:

$ journalctl -u apache.service

-- Logs begin at Fri 2013-12-13 23:43:32 UTC, end at Sun 2013-12-22 12:32:52 UTC. --
Dec 22 12:32:39 localhost systemd[1]: Starting Apache Service...
Dec 22 12:32:39 localhost systemd[1]: Started Apache Service.
Dec 22 12:32:39 localhost docker[9772]: /usr/sbin/apache2ctl: 87: ulimit: error setting limit (Operation not permitted)
Dec 22 12:32:39 localhost docker[9772]: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.6 for ServerName

Using the --tunnel flag (docs), you can remotely read the journal for a specific unit started via fleet. This command will figure out which machine the unit is currently running on, fetch the journal and output it:

$ fleetctl --tunnel 10.10.10.10 journal apache.service

-- Logs begin at Fri 2013-12-13 23:43:32 UTC, end at Sun 2013-12-22 12:32:52 UTC. --
Dec 22 12:32:39 localhost systemd[1]: Starting Apache Service...
Dec 22 12:32:39 localhost systemd[1]: Started Apache Service.
Dec 22 12:32:39 localhost docker[9772]: /usr/sbin/apache2ctl: 87: ulimit: error setting limit (Operation not permitted)
Dec 22 12:32:39 localhost docker[9772]: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.6 for ServerName

Read entries since boot

Reading just the entries since the last boot is an easy way to troubleshoot services that are failing to start properly:

journalctl --boot

Tail the journal

You can tail the entire journal or just a specific service:

journalctl -f
journalctl -u apache.service -f

Read entries with line wrapping

By default journalctl passes FRSXMK command line options to less. You can override these options by setting a custom SYSTEMD_LESS environment variable with omitted S option:

SYSTEMD_LESS=FRXMK journalctl

Read logs without pager:

journalctl --no-pager

Debugging journald

If you've faced some problems with journald you can enable debug mode following the instructions below.

Enable debugging manually

mkdir -p /etc/systemd/system/systemd-journald.service.d/

Create Drop-In /etc/systemd/system/systemd-journald.service.d/10-debug.conf with following content:

[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

And restart systemd-journald service:

systemctl daemon-reload
systemctl restart systemd-journald
dmesg | grep systemd-journald

Enable debugging via cloud-config

Define a Drop-In in a Cloud-Config:

#cloud-config
coreos:
  units:
    - name: systemd-journald.service
      drop-ins:
        - name: 10-debug.conf
          content: |
            [Service]
            Environment=SYSTEMD_LOG_LEVEL=debug
      command: restart
CoreOS cloud-configs can validated using the online validator.

And run coreos-cloudinit or reboot your Container Linux host to apply the changes.

Enable debugging via Ignition

Define a Drop-In in an Ignition config:

{
  "ignition": { "version": "2.0.0" },
  "systemd": {
    "units": [{
      "name": "systemd-journald.service",
      "dropins": [{
        "name": "10-debug.conf",
        "contents": "[Service]\nEnvironment=SYSTEMD_LOG_LEVEL=debug"
      }]
    }]
  }
}

More information

Getting Started with systemd Network Configuration with networkd