RHEL7: How to get started with Systemd.

Presentation

As the Systemd now replaces SysVinit, it is time to get familiar with it and learn new commands. You can use this page as a Systemd tutorial.

Systemd makes a server boot quicker because it uses fewer scripts and tries to run more tasks in parallel, Systemd calls them units (to get a better understanding about the problems Systemd solves, read this thread from an ArchLinux maintainer).

The global Systemd configuration is stored in the /etc/systemd directory.
Service configuration files are located in the /usr/lib/systemd/system directory.
Custom service configuration files are stored in the /etc/systemd/system directory.
The /run/systemd/system directory is also used (see the reason why the /run directory was created here).

To get the current release of Systemd, type:

# systemctl --version
systemd 208
+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ

Note: The RHEL 7.2 release brings an upgrade of Systemd to version 219 (already available for pre-RHEL 7.2 here).

# systemctl --version
systemd 219
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN

Boot Process

Systemd primary task is to manage the boot process and provides information about it.
To get the boot process duration, type:

# systemd-analyze
Startup finished in 422ms (kernel) + 2.722s (initrd) + 9.674s (userspace) = 12.820s

To get the time spent by each task during the boot process, type:

# systemd-analyze blame
7.029s network.service
2.241s plymouth-start.service
1.293s kdump.service
1.156s plymouth-quit-wait.service
1.048s firewalld.service
632ms postfix.service
621ms tuned.service
460ms iprupdate.service
446ms iprinit.service
344ms accounts-daemon.service
...
7ms systemd-update-utmp-runlevel.service
5ms systemd-random-seed.service
5ms sys-kernel-config.mount

Seth Jennings created a short video about the systemd-analyze blame command.

To get the list of the critical chain of tasks during the boot process (any additional delay of these tasks would cause an increase of the overall boot time), type:

# systemd-analyze critical-chain
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

multi-user.target @6.167s
└─mariadb.service @2.661s +3.505s
  └─network.target @2.649s
    └─network.service @2.168s +478ms
      └─NetworkManager.service @1.993s +174ms
        └─firewalld.service @826ms +1.162s
          └─basic.target @822ms
            └─sockets.target @821ms
              └─dbus.socket @820ms
                └─sysinit.target @813ms
                  └─systemd-update-utmp.service @806ms +6ms
                    └─auditd.service @747ms +58ms
                      └─local-fs.target @742ms
                        └─boot.mount @709ms +32ms
                          └─systemd-fsck@dev-disk-by\x2duuid-497a43ab\x2d33b0\x2
                            └─dev-disk-by\x2duuid-497a43ab\x2d33b0\x2d4153\x2d9f

To get the list of the critical chain for a particular service (here firewalld), type:

# systemd-analyze critical-chain firewalld.service
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

firewalld.service +1.162s
└─basic.target @822ms
  └─sockets.target @821ms
    └─dbus.socket @820ms
      └─sysinit.target @813ms
        └─systemd-update-utmp.service @806ms +6ms
          └─auditd.service @747ms +58ms
            └─local-fs.target @742ms
              └─boot.mount @709ms +32ms
                └─systemd-fsck@dev-disk-by\x2duuid-497a43ab\x2d33b0\x2d4153\x2d9
                  └─dev-disk-by\x2duuid-497a43ab\x2d33b0\x2d4153\x2d9f8d\x2d7788

To get the list of the critical chain for a particular target (here basic.target), type:

# systemd-analyze critical-chain basic.target | grep target
basic.target @822ms
└─sockets.target @821ms
    └─sysinit.target @813ms
          └─local-fs.target @742ms

To get the list of the dependencies, type:

# systemctl list-dependencies
default.target
├─abrt-ccpp.service
├─abrt-oops.service
...
├─tuned.service
├─basic.target
│ ├─firewalld.service
│ ├─microcode.service
...
├─getty.target
│ ├─getty@tty1.service
│ └─serial-getty@ttyS0.service
└─remote-fs.target

To get the list of dependencies for a particular service (here sshd), type:

# systemctl list-dependencies sshd.service

To get the list of dependencies for a particular target (here graphical.target), type:

# systemctl list-dependencies graphical.target | grep target
graphical.target
└─multi-user.target
  ├─basic.target
  │ ├─paths.target
  │ ├─slices.target
  │ ├─sockets.target
  │ ├─sysinit.target
  │ │ ├─cryptsetup.target
  │ │ ├─local-fs.target
  │ │ └─swap.target
  │ └─timers.target
  ├─getty.target
  └─remote-fs.target

You can find additional information on this point in the Lennart Poettering’s blog.
In case you get boot problems, the Systemd debugging page may also help you.

Journal Analysis

In addition, Systemd handles the system event log through Journald.
The reasons behind Journald creation are explained in this Introduction to Journald.
The rsyslogd daemon, although present by default, is not mandatory any more.

To get the content of the Systemd journal, type:

# journalctl

Note: It is possible to add a specific user to the systemd-journal group to view journals without switching to root.

To get all the events related to the crond process in the journal, type:

# journalctl /sbin/crond

Note: You can replace /sbin/crond by `which crond`.

To get all the events related to a specific service (here chronyd), type:

# journalctl --unit=chronyd

Note1: You can also type: # journalctl -u chronyd
Note2: Starting with RHEL 7.2, you can now use a command like journalctl –unit=c*.

To get all the events since the last boot, type:

# journalctl -b

To get all the events that appeared today in the journal, type:

# journalctl --since=today

To get all the events with a syslog priority of err, type:

# journalctl -p err

To get the 10 last events and wait for any new one (like “tail -f /var/log/messages“), type:

# journalctl -f

With the RHEL 7.2 release, to get the list of the recent boots, type:

# journalctl --list-boots

By default, Journald logs are stored in the /run/log/journal directory and disappear after a reboot.
To store Journald logs in a more permanent way, type:

# mkdir /var/log/journal
# echo "SystemMaxUse=50M" >> /etc/systemd/journald.conf
# systemctl restart systemd-journald 

Note1: Setting the SystemMaxUse variable is necessary because otherwise 10% of the filesystem where the /var/log/journal directory is stored may be used at maximum by the journal. Besides the space used, this will slow down the boot process and potentially the whole system (see details here). Furthermore, when using small VMs, the size of the journal can become critical (see this article about the journal size and the Storage parameter).
Note2: Starting with Systemd v219 and RHEL 7.2, the filesystem where the /var/log/journal directory is located requires ACL support (see details here)

To display the disk space used by Journald at any time, type:

# journalctl --disk-usage
Journals take up 8.0M on disk.

Note: With RHEL 7.2 and the 219 Systemd version, two new options have been added: –vacuum-size=1M and –vacuum-time=1day. They reduce the space used by archived journal files respectively to 1 MB or to the last 24 hours. These restrictions may be combined. They only operate on archived, not on active journal files.

Additional information about Journald is available in the Lennart Poettering’s blog or Lennart Poettering’s video (44min: the first ten minutes are very interesting concerning security issues).
Fedora Magazine also published an interesting article, Systemd: Using the journal.
Rainer Gerhards, rsyslog main author, discusses the arguments around the creation of Journald.
Nikolai Bezroukov wrote an authoritative page about Syslog.

Control Groups

Systemd organizes processes in control groups. For example, all the processes started by an apache webserver will be in the same control group, CGI scripts included.

To get the full hierarchy of control groups, type:

# systemd-cgls
├─user.slice
│ └─user-1000.slice
│ └─session-1.scope
│ ├─2889 gdm-session-worker [pam/gdm-password]
│ ├─2899 /usr/bin/gnome-keyring-daemon --daemonize --login
│ ├─2901 gnome-session --session gnome-classic
. .
└─iprupdate.service
└─785 /sbin/iprupdate --daemon

To get the list of control group ordered by CPU, memory and disk I/O load, type:

# systemd-cgtop
Path Tasks %CPU Memory Input/s Output/s
/ 213 3.9 829.7M - -
/system.slice 1 - - - -
/system.slice/ModemManager.service 1 - - - -

To kill all the processes associated with an apache server (CGI scripts included), type:

# systemctl kill httpd

To put resource limits on a service (here 500 CPUShares), type:

# systemctl set-property httpd.service CPUShares=500

Note1: The change is written into the service unit file. Use the –runtime option to avoid this behavior.
Note2: By default, each service owns 1024 CPUShares. Nothing prevents you from giving a value smaller or bigger.

To get the current CPUShares service value, type:

# systemctl show -p CPUShares httpd.service
CPUShares=500

Or:

# systemctl show httpd.service | grep CPUShares
CPUShares=500

Go to the CGroup page for a more detailed discussion on this topic.

Sources: New control group interface, Systemd 205 announcement.

Service Management

Systemd deals with all the aspects of the service management. The systemctl command replaces the chkconfig and the service commands. The old commands are now a link to the systemctl command.

To activate the NTP service at boot, type:

# systemctl enable ntpd

Note1: You should specify ntpd.service but by default the .service suffix will be added.
Note2: If you specify a path, the .mount suffix will be added.
Note3: If you mention a device, the .device suffix will be added.

To deactivate it, start it, stop it, restart it, reload it, type:

# systemctl disable ntpd
# systemctl start ntpd
# systemctl stop ntpd
# systemctl restart ntpd
# systemctl reload ntpd

Note1: The reload option is not available for all the services.
Note2: It is also possible to mask and unmask a service. Masking a service prevents it from being started manually, by another service or by D-Bus.

To know if the NTP service is activated at boot, type:

# systemctl is-enabled ntpd
enabled

To know if the NTP service is running, type:

# systemctl is-active ntpd
inactive

To get the status of the Apache service, type:

# systemctl status httpd
httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled)
   Active: active (running) since Tue 2014-08-21 11:48:23 CEST; 44s ago
 Main PID: 2446 (httpd)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /system.slice/httpd.service
           └─2446 /usr/sbin/httpd -DFOREGROUND
           └─2447 /usr/sbin/httpd -DFOREGROUND
           └─2448 /usr/sbin/httpd -DFOREGROUND
           └─2449 /usr/sbin/httpd -DFOREGROUND
           └─2450 /usr/sbin/httpd -DFOREGROUND
           └─2451 /usr/sbin/httpd -DFOREGROUND
           └─2452 /usr/sbin/httpd -DFOREGROUND

Aug 21 11:48:23 server.example.com systemd[1]: Starting The Apache HTTP Ser....
Aug 21 11:48:23 server.example.com systemd[1]: Started The Apache HTTP Server.
Hint: Some lines were ellipsized, use -l to show in full.

Note1: In the line starting with “Loaded:”, you learn that the unit configuration file named /usr/lib/systemd/system/httpd.service has been loaded by Systemd. You also learn that the service isn’t started at boot time (disabled).
Note2: In the line beginning by “Active:”, you are informed that the service has been started for 44 seconds and is thus currently running (active).
Note3: The “Status:” line is specific to the Apache service and gives you a quick report of the service activity.
Note4: The line starting with “CGroup:” gives you all the information about the CGroup that the Apache service is part of (here /system.slice/httpd.service).

If you change a service configuration, you will need to reload it:

# systemctl daemon-reload

Note: The systemd-delta command displays the list of changes in the service configuration files.

To get the list of all the units (services, mount points, devices) with their status and description, type:

# systemctl

Note: You can use the arrow keys to move inside the unit list.

To get a more readable list, type:

# systemctl list-unit-files

To get the list of services that failed at boot, type:

# systemctl --failed

To get the status of a service (here httpd) on a remote server (here rhel7.example.com), type:

# systemctl -H root@rhel7.example.com status httpd.service

The RHEL 7.2 release brings the ability to connect as root to a container inside a host.
To get the status of a service (here httpd) in a container (here httpsrv) on a remote server (here rhel72.example.com), type:

# systemctl -H root@rhel72.example.com:httpsrv status httpd.service

Note: This feature is only available for the root user, as entering containers is a privileged operation.

To get all the configuration details about a service (here httpd), type:

# systemctl show httpd
Id=httpd.service
Names=httpd.service
Requires=basic.target
Wants=system.slice
Conflicts=shutdown.target
Before=shutdown.target
After=network.target remote-fs.target nss-lookup.target systemd-journald.socket
Description=The Apache HTTP Server
LoadState=loaded
ActiveState=inactive
...
ExecMainPID=0
ExecMainCode=0
ExecMainStatus=0

Digital Ocean provides an excellent article about Systemd unit files.
Alexander Patrakov‘s blog also provides a very good introduction for writing Systemd service files.
In addition, the FedoraProject website can also help you if you need to write Systemd unit files.
Finally, FedoraMagazine published a great article about systemd unit dependencies and order.

The RHEL 7.2 release brings the systemctl cat command to display a unit file.
To display a service configuration file (here rngd), type:

# systemctl cat rngd
# /usr/lib/systemd/system/rngd.service
[Unit]
Description=Hardware RNG Entropy Gatherer Daemon

[Service]
ExecStart=/sbin/rngd -f

[Install]
WantedBy=multi-user.target

Note: The systemctl cat rngd command carries several advantages compared to a simple cat rngd.service command:

• You don’t need to know which version is executed (the default version in /usr/lib/system/system/rngd.service or a customized version in /etc/systemd/system/rngd.service),
• You get at the top the current version starting with a # character,
• If there are “drop-in” unit file snippets (here in /etc/systemd/system/rngd.service.d), they are inserted at the right place.

The RHEL 7.2 release also adds the systemctl edit command.
By default, this command edits a drop-in snippet called override.conf or creates it if it doesn’t already exist.
After the snippet has been edited, systemd configuration is reloaded.
For example, to edit/create a drop-in file for the httpd service, type:

# systemctl edit httpd

Note1: The /etc/systemd/system/httpd.service.d directory is automatically created if necessary.
Note2: Editing is done with an editor specified by the SYSTEMD_EDITOR, EDITOR or VISUAL environment variables in this order. By default, the nano, vim, and vi editors will be tried one after the other. If like me you prefer vim to nano, don’t forget to type export EDITOR=vim or to set up your environment before executing systemctl edit.
Note3: No need to type systemctl daemon-reload, systemctl edit takes care of it.

If you add the –full option, it is no longer a drop-in snippet that is edited or created, it’s the unit file itself.
To modify the content of the httpd unit file, type:

# systemctl edit httpd --full

Note1: Be careful when editing this way, your changes will be active right after exiting the editor.
Note2: You don’t need to specify where the unit file is located.
Note3: Instead of using the –full option, you can use the –runtime option to edit a unit file if no /etc/systemd/system unit file exists for a given service and you don’t want the change to survive a reboot.

Targets/Run Levels

Systemd also deals with targets. A target is a grouping mechanism allowing several services to start at the same time. The list of services associated with a particular target is stored in a directory named with the target name and the suffix “.wants”. In some way, targets replace run levels but they are more general. To move to maintenance mode, type:

# systemctl rescue

Note: There is also an emergency target only available when set up in the kernel boot line (systemd.unit=emergency.target) and for critical situations.

To move to the level 3 (equivalent to the previous level 3), type:

# systemctl isolate runlevel3.target

Or

# systemctl isolate multi-user.target

To move to the graphical level (equivalent to the previous level 5), type:

# systemctl isolate graphical.target

To set the default run level to non-graphical mode, type:

# systemctl set-default multi-user.target

To set the default run level to graphical mode, type:

# systemctl set-default graphical.target

To get the current default run level, type:

# systemctl get-default
graphical.target

To stop a server, type:

# systemctl poweroff

Note: You can still use the poweroff command, a link to the systemctl command has been created (the same thing is true for the halt and reboot commands).

To reboot a server, suspend it or put it into hibernation, type:

# systemctl reboot
# systemctl suspend
# systemctl hibernate

In case you get boot or shutdown problems, the Systemd debugging page may help you.

Linux Standardization

Systemd‘s authors have decided to help Linux standardization among distributions. Through Systemd, changes happen in the localization of some configuration files.

Miscellaneous

To get the server hostnames, type:

# hostnamectl
   Static hostname: rhel7.example.com
         Icon name: computer-laptop
           Chassis: laptop
        Machine ID: bcdc71f1943f4d859aa37e54a422938d
           Boot ID: f84556924b4e4bbf9c4a82fef4ac26d0
  Operating System: Red Hat Enterprise Linux Everything 7.0 (Maipo)
       CPE OS Name: cpe:/o:redhat:enterprise_linux:7.0:beta:everything
            Kernel: Linux 3.10.0-54.0.1.el7.x86_64
      Architecture: x86_64

Note: There are three kinds of hostnames: static, pretty, and transient.
“The static host name is the traditional hostname, which can be chosen by the user, and is stored in the /etc/hostname file. The transient hostname is a dynamic host name maintained by the kernel. It is initialized to the static host name by default, whose value defaults to localhost. It can be changed by DHCP or mDNS at runtime. The pretty hostname is a free-form UTF8 host name for presentation to the user.” Source: RHEL 7 Networking Guide.

To assign the rhel7 hostname permanently to the server, type:

# hostnamectl set-hostname rhel7

Note: With this syntax all three hostnames (static, pretty, and transient) take the rhel7 value at the same time. However, it is possible to set the three hostnames separately by using the –pretty, –static, and –transient options.

To get the current locale, virtual console keymap and X11 layout, type:

# localectl
System Locale: LANG=en_US.UTF-8
VC Keymap: en_US
X11 Layout: en_US

To assign the en_GB.utf8 value to the locale, type:

# localectl set-locale LANG=en_GB.utf8

To assign the en_GB value to the virtual console keymap, type:

# localectl set-keymap en_GB

To assign the en_GB value to the X11 layout, type:

# localectl set-x11-keymap en_GB

To get the current date and time, type:

# timedatectl
Local time: Fri 2014-01-24 22:34:05 CET
Universal time: Fri 2014-01-24 21:34:05 UTC
RTC time: Fri 2014-01-24 21:34:05
Timezone: Europe/Madrid (CET, +0100)
NTP enabled: yes
NTP synchronized: yes
RTC in local TZ: no
DST active: no
Last DST change: DST ended at
Sun 2013-10-27 02:59:59 CEST
Sun 2013-10-27 02:00:00 CET
Next DST change: DST begins (the clock jumps one hour forward) at
Sun 2014-03-30 01:59:59 CET
Sun 2014-03-30 03:00:00 CEST

To set the current date, type:

# timedatectl set-time YYYY-MM-DD

To set the current time, type:

# timedatectl set-time HH:MM:SS

To get the list of time zones, type:

# timedatectl list-timezones

To change the time zone to America/New_York, type:

# timedatectl set-timezone America/New_York

To get the users’ list, type:

# loginctl list-users
UID USER
42 gdm
1000 tom
0 root

To get the list of all current user sessions, type:

# loginctl list-sessions
SESSION UID USER SEAT
1 1000 tom seat0

1 sessions listed.

To get the properties of the user tom, type:

# loginctl show-user tom
UID=1000
GID=1000
Name=tom
Timestamp=Fri 2014-01-24 21:53:43 CET
TimestampMonotonic=160754102
RuntimePath=/run/user/1000
Slice=user-1000.slice
Display=1
State=active
Sessions=1
IdleHint=no
IdleSinceHint=0
IdleSinceHintMonotonic=0

Additional Resources

For a better understanding, you can additionally read:

• a detailed introduction to Systemd,
• Bob Cromwell’s blog about Systemd,
• an overview of Systemd by Red Hat,
• a first dive into Systemd by Etsuji Nakai,
• a Systemd presentation by Ben Breard,
• an interview of Lennart Poettering explaining the story behind Systemd,
• the 2015 Red Hat Summit Systemd presentation talking about version 219 new features,
• an article about converting Sysvinit scripts into Systemd unit files,
• a description of the Systemd bug fix and enhancement updates coming with the version 219,
• the Systemd changelog.

You can also watch a video about Working with systemd by Sander van Vugt (43min/2015).

https://www.certdepot.net/rhel7-get-started-systemd/