Systemd

General

• systemd is a "system and service manager" for Linux operating systems
• "An init system used to bootstrap user space and manage user proceses"
  • When run as first process on boot (as PID 1), it acts as init system that brings up and maintains userspace services
  • bootstrap ("booting") - a self-starting process that is supposed to proceed without external input
• Systemd knows various kinds of dependencies for the purposes of booting, for example
  • Positive and negative requirement dependencies (i.e. Requires= and Conflicts=)
  • Ordering dependencies (After= and Before=)
  • If required but not ordered they can start in parallel

Things systemd does

• init daemon
• device management
• login management
• network connection management
• event logging

Benefits

• Express dependencies
• parallel processing during boot
• If you have an app and you need something to monitor it

Systemd Utilities

• systemctl
• journalctl
• loginctl
• cgtop
• notify
• analyze
• cgls
• nspawn

Systemd Daemons

• systemd
• journald - event-logging
• logind - manages user logins
• networkd - handle the configuration of network daemons
• tmpfiles - creation and cleanup of temporary files and directories
• timedated
• udevd - device manager for the Linux kernel which handles the /dev directory and all user space actions when adding/removing devices
• systemd-boot - a simpler GRUB alternative. Uses split configuration files for each given OS available, ergo simpler to maintain. GRUB is like it's own OS.

Systemd Units

• Systemd provides a dependency system between various entities called "units" of 12 different types
• Units encapsulate various objects that are relevant for system boot-up and maintenance
• man systemd.unit shows common options for all unit configuration files

Service unit

• Start and control daemons and the processes they consist of
• man systemd.service

Socket unit

• Encapsulate local IPC or network sockets in the system
• Useful for socket-based activation
• man systemd.socket

Target units

• Unit groups
• man systemd.target

Device units

• Expose kernel devices in systemd and may be used to implement device-based activation
• man systemd.device

Mount units

• Control mount points in the file system
• man systemd.mount

Automount units

• On-demand mounting of file systems as well as parallelized boot-up
• man systemd.automount

Timer units

• Cron replacement
• man systemd.timer

Path units

• May be used to activate other services when file system objects change or are modified
• man systemd.path

Slice units

• Group units which manage system processes (such as service and scope units) in a hierarchical tree for resource management purposes

Scope units

• Similar to service units, but manage foreign processes instead of starting them as well

Systemd Configuration files

• The main configuration file is read before any of the configuration directories, and has the lowest precedence

• It is recommended to prefix all filenames in those subdirectories with a two-digit number and a dash, to simplify the ordering of the files

System conf files

• When systemd is run as a system instance, these files are read:

1. /etc/systemd/system.conf
2. /etc/systemd/system.conf.d/*.conf
3. /run/systemd/system.conf.d/*.conf
4. /usr/lib/systemd/system.conf.d/*.conf

User conf files

1. /etc/systemd/user.conf
2. /etc/systemd/user.conf.d/*.conf
3. /run/systemd/user.conf.d/*.conf
4. /usr/lib/systemd/user.conf.d/*.conf

systemctl

• Control the systemd system and service manager
• Used to introspect and control the state of the "systemd" system and service manager.

Usage

Hierarchical view of currently operating services

• sudo systemctl status
  • You'll see your user's "slice" of system processes
  • Then there's the system's "slice" of processes. Pretty much everything will be under there, like sshd.service, nginx.service, mariadb.service
• systemctl daemon-reload - run this after you put a new .service file into /usr/local/lib/systemd/system

Full list of loaded and inactive services

• systemctl list-units
• Shows color-coded statuses
• Shows could be what triggers system errors to be reported to IT

Start a service on startup/restart

• systemctl enable/disable nginx

journalctl

• query the systemd journal as written by systemd-journald.service
• journalctl - called w/o params lists everything

journal field types

• USER journal fields
  • MESSAGE=
  • MESSAGE_ID=
  • PRIORITY=
  • CODE_FILE=, CODE_LINE=, CODE_FUNC=
  • ERRNO=
  • others...
• TRUSTED journal fields, prefixed with an underscore, are implicitly added to the journal and cannot be altered by client code
  • _PID= _UID=, _GID=
  • _COMM= - name
  • _EXE= - executable path
  • _CMD_LINE= - command line of the process the journal entry originates from
  • _SYSTEMD_* - systemd hierarchy info
  • SELINUX_CONTEXT=
  • others ..
• KERNEL journal fields

Creating your own unit

• systemd unit file creating a service - good info in here
• Gunicorn as a systemd service
• Service file format is like Windows .ini

Service file names/directories

• Typical extensions are *.socket, *.target, *.service
• If installed by packages/package manager
  • /usr/lib/systemd/system/
  • /usr/lib/systemd/*.conf.d/
• Software I develop goes in local
  • /usr/local/lib/systemd/system
• Override config installed by existing packages
  • /etc/systemd/system
  • /etc reserved for the local administrator in general

.service file sections

Unit

• man systemd.unit for possible entries in the [Unit] section
• Description= - single line that appears in service list and logs
• Documentation=https://online.docs.com
• WantedBy= - when to start this service
  • multi-user.target - when server is OK and is ready to run command line applications
  • graphical.target - when GNOME or KDE is ready
  • network-up.target - when server is connected properly to a network
• Requires=nginx.service docker.socket - examples
• Wants= - a weaker version of requires
• After=network.target
• Before=
• Conflicts=
• AssertPathExists=/srv/webserver
• StartLimitIntervalSec=0 - Place no limit on the number of tries systemd attempts to restart the service if it fails.

Service

• Section is required for services
• Type=
  • simple - The executable won't fork other processes
  • forking - The executable will fork other processes
  • others: oneshot, dbus, notify, idle
• ExecStart=
  • Use absolute paths
  • No need to start a shell like the old days
• ExecStop= - exists
• ExecReload=/bin/kill -USR2 $MAINPID - example from php-fpm.service
• ExecReload=/bin/kill -s HUP $MAINPID - example from nginx.service
• KillSignal=SIGQUIT
• Restart=
  • always - keep restarting it whenever it terminates/ crashes, or until you do systemctl stop service-name.service.
  • on-abnormal - restart on crash only
  • on-failure - restart when exit code is non-zero.
  • no - don't restart automatically
• RestartSec=1
• User=
• Group=
• Environment=PATH=/usr/local/bin/:$PATH - set environment variable right in the service file
• EnvironmentFile=/etc/sysconfig/php-fpm - example from php-fpm.service
• PrivateTmp=yes
• WorkingDirectory=
• RuntimeDirectory=php-fpm
• RuntimeDirectoryMode=755

Exec

• defines the execution environment the commands are executed in

Kill

• defines the way the processes of the service are terminated

Examples

An example .service file for a web app

[Unit]
Description=Web Application HTTP server (running in port 8080)
WantedBy=multi-user.target

[Service]
Type=simple
ExecStart=/usr/bin/python3 /usr/local/bin/penguin-web-app/main.py
Restart=always

[Install]
WantedBy=multi-user.target

.socket unit file

[Unit]
Description=Docker Socket for the API
PartOf=docker.service

[Socket]
ListenStream=/var/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker

[Install]
WantedBy=sockets.target

Gunicorn Web API backend

# CircInteractome  client server
# Author: Chris Coletta <christopher.coletta@nih.gov>, Github: colettace

# This is a systemd "unit" file, which controls how the CircInteractome server daemon is created
# on system reboot, and how it should be respawn after a crash of the app

# httpd service must be enabled. Install the Apache Web Server and run this command:
# $ sudo systemctl enable httpd
# Expected output:
# Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.
# Confirm that it's enabled by running this command:
# $ sudo systemctl status
# Check to see it's slotted under the "system slice"

# Requires a user/group. Make up a user/group id that's greater than 1000
# sudo groupadd -g <a number> <a group name>
# useradd -c "CircInteractome daemon owner created by colettace" -g <the group name> -s /bin/false -M -u <the number> <a new unix username>

# How to install this file and enable it:
# sudo mkdir -p /usr/local/lib/systemd/system
# sudo cp ./circinteractome.service /usr/local/lib/systemd/system
# sudo systemctl enable circinteractome.service

[Unit]
Description=CircInteractome backend server (API created by Python's FastAPI package, running within a Gunicorn server)
WantedBy=network-up.target
Requires=httpd.service
After=httpd.service
StartLimitIntervalSec=0
AssertPathExists=/home/colettace/circinteractome/server

[Service]
Type=simple
# Uncomment this for production
# Type=forking
#PermissionsStartOnly=true
User=circinteractome
Group=circinteractome
WorkingDirectory=/home/colettace/circinteractome/server
ExecStart = /usr/local/bin/uvicorn circinteractome_api_v2:outer_shell_app --proxy-headers --log-level debug --workers 4
# For production, something like this should be sufficient
# # The following hasn't been tested yet!!
#ExecStart =  /usr/local/bin/gunicorn circinteractome_server_backend:outer_shell_app -w 4 -k uvicorn.workers.UvicornWorker  --proxy-headers -uds /var/run/gunicorn/gunicorn.sock
ExecReload = /bin/kill -s HUP $MAINPID
ExecStop = /bin/kill -s TERM $MAINPID