Reading the system log

journalctl is your interface into a single machine's journal/logging and fleetctl journal will fetch the journal for containers started with fleet. All service files and Docker containers insert data into the systemd journal. There are a few helpful commands to read the journal:

Read the entire journal

$ journalctl

-- Logs begin at Fri 2013-12-13 23:43:32 UTC, end at Sun 2013-12-22 12:28:45 UTC. --
Dec 22 00:10:21 localhost systemd-journal[33]: Runtime journal is using 184.0K (max 49.9M, leaving 74.8M of free 499.0M, current limit 49.9M).
Dec 22 00:10:21 localhost systemd-journal[33]: Runtime journal is using 188.0K (max 49.9M, leaving 74.8M of free 499.0M, current limit 49.9M).
Dec 22 00:10:21 localhost kernel: Initializing cgroup subsys cpuset
Dec 22 00:10:21 localhost kernel: Initializing cgroup subsys cpu
Dec 22 00:10:21 localhost kernel: Initializing cgroup subsys cpuacct
Dec 22 00:10:21 localhost kernel: Linux version 3.11.7+ (buildbot@10.10.10.10) (gcc version 4.6.3 (Gentoo Hardened 4.6.3 p1.13, pie-0.5.2)
...
1000s more lines

Read entries for a specific service

Read entries generated by a specific unit:

$ journalctl -u apache.service

-- Logs begin at Fri 2013-12-13 23:43:32 UTC, end at Sun 2013-12-22 12:32:52 UTC. --
Dec 22 12:32:39 localhost systemd[1]: Starting Apache Service...
Dec 22 12:32:39 localhost systemd[1]: Started Apache Service.
Dec 22 12:32:39 localhost docker[9772]: /usr/sbin/apache2ctl: 87: ulimit: error setting limit (Operation not permitted)
Dec 22 12:32:39 localhost docker[9772]: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.6 for ServerName

Using the --tunnel flag (docs), you can remotely read the journal for a specific unit started via fleet. This command will figure out which machine the unit is currently running on, fetch the journal and output it:

$ fleetctl --tunnel 10.10.10.10 journal apache.service

-- Logs begin at Fri 2013-12-13 23:43:32 UTC, end at Sun 2013-12-22 12:32:52 UTC. --
Dec 22 12:32:39 localhost systemd[1]: Starting Apache Service...
Dec 22 12:32:39 localhost systemd[1]: Started Apache Service.
Dec 22 12:32:39 localhost docker[9772]: /usr/sbin/apache2ctl: 87: ulimit: error setting limit (Operation not permitted)
Dec 22 12:32:39 localhost docker[9772]: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.6 for ServerName

Read entries since boot

Reading just the entries since the last boot is an easy way to troubleshoot services that are failing to start properly:

journalctl --boot

Tail the journal

You can tail the entire journal or just a specific service:

journalctl -f
journalctl -u apache.service -f

Read entries with line wrapping

By default journalctl passes FRSXMK command line options to less. You can override these options by setting a custom SYSTEMD_LESS environment variable with omitted S option:

SYSTEMD_LESS=FRXMK journalctl

Read logs without pager:

journalctl --no-pager

Debugging journald

If you've faced some problems with journald you can enable debug mode following the instructions below.

Enable debugging manually

mkdir -p /etc/systemd/system/systemd-journald.service.d/

Create Drop-In /etc/systemd/system/systemd-journald.service.d/10-debug.conf with following content:

[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

And restart systemd-journald service:

systemctl daemon-reload
systemctl restart systemd-journald
dmesg | grep systemd-journald

Enable debugging via a Container Linux Config

Define a Drop-In in a Container Linux Config:

This is the human-readable config file. This should not be immediately passed to Container Linux. Learn more.
# This config is meant to be consumed by the config transpiler, which will
# generate the corresponding Ignition config. Do not pass this config directly
# to instances of Container Linux.

systemd:
  units:
    - name: systemd-journald.service
      dropins:
        - name: 10-debug.conf
          contents: |
            [Service]
            Environment=SYSTEMD_LOG_LEVEL=debug
This is the raw machine configuration, which is not intended for editing. Learn more. Validate the config here.
{
  "ignition": {
    "version": "2.0.0",
    "config": {}
  },
  "storage": {},
  "systemd": {
    "units": [
      {
        "name": "systemd-journald.service",
        "dropins": [
          {
            "name": "10-debug.conf",
            "contents": "[Service]\nEnvironment=SYSTEMD_LOG_LEVEL=debug"
          }
        ]
      }
    ]
  },
  "networkd": {},
  "passwd": {}
}

More information

Getting Started with systemd Network Configuration with networkd