English
A hardened systemd service
Step 1. Why systemd
Why. The API is a long-running process. Started by hand in an SSH session, it would stop when the session closes, would not come back after a crash or a reboot, and its output would be lost. Tools like nohup or screen keep it alive but solve none of the other problems.
systemd is the service manager of Ubuntu: process number 1, which starts and supervises everything else. A service is described declaratively in a unit file, and systemd takes care of the rest:
- starts the service at boot, in the right order after the network and the database;
- runs it as the configured user, with the configured environment;
- restarts it when it crashes;
- collects everything it writes to the console in the journal, read with
journalctl; - and — the part that is easy to overlook — puts it into a sandbox made of Linux kernel features, configured with a few lines.
Docker would be the other common choice. It gives similar isolation, but adds an image build, a registry and a container runtime to the deployment; for one .NET process on one server, systemd does the job with what Ubuntu already has.
Step 2. The unit file
Do. On your computer, create the unit in the folder Deploy/systemd:
ini
# Chatex API (Kestrel on 127.0.0.1:5000, behind Nginx).
# Install: cp Deploy/systemd/chatex-api.service /etc/systemd/system/ && systemctl daemon-reload
# systemctl enable chatex-api
# Logs: journalctl -u chatex-api -f
[Unit]
Description=Chatex API
After=network-online.target postgresql.service
Wants=network-online.target
[Service]
Type=exec
User=chatex
Group=chatex
# /opt/chatex/current is a symlink to the active release; it is resolved on every (re)start.
WorkingDirectory=/opt/chatex/current/api
ExecStart=/usr/bin/dotnet /opt/chatex/current/api/Chatex.Api.dll
EnvironmentFile=/etc/chatex/chatex.env
Environment=ASPNETCORE_ENVIRONMENT=Production
Environment=ASPNETCORE_URLS=http://127.0.0.1:5000
Environment=DOTNET_NOLOGO=true
Environment=DOTNET_CLI_TELEMETRY_OPTOUT=true
Restart=always
RestartSec=5
KillSignal=SIGINT
TimeoutStopSec=30
SyslogIdentifier=chatex-api
# Hardening: the release is read-only; only /var/lib/chatex (uploads, Data Protection keys) is writable.
# AF_NETLINK lets .NET read network interface information; nothing else beyond TCP and Unix sockets is allowed.
NoNewPrivileges=true
PrivateTmp=true
PrivateDevices=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=/var/lib/chatex
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictSUIDSGID=true
LockPersonality=true
ProtectClock=true
RestrictNamespaces=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
CapabilityBoundingSet=
UMask=0027
[Install]
WantedBy=multi-user.target[Unit]
Descriptionis the name shown bysystemctl status.After=network-online.target postgresql.serviceorders the start after the network is up and PostgreSQL has started;Wants=network-online.targetmakes sure the network target is actually reached at boot.Afteronly orders, it does not create a dependency: the API also starts when PostgreSQL is down, and then reportsUnhealthyon/healthuntil the database is back (topic 13).
[Service]
Type=exec— the service counts as started as soon as thedotnetprogram has been executed; a failure to execute it (a wrong path) is reported bysystemctlat once.User=chatex,Group=chatex— the user of the previous chapter.WorkingDirectoryandExecStartrunChatex.Api.dllfrom/opt/chatex/current/api, with thedotnethost of the runtime package.currentis the symbolic link to the active release; it is resolved again on every start, so after the link has been switched,systemctl restartstarts the new version. The unit file never changes between releases.EnvironmentFile=/etc/chatex/chatex.envloads the settings of the previous chapter. systemd reads the file asroot, before it switches tochatex.Environment=ASPNETCORE_ENVIRONMENT=Productionselects the production environment:appsettings.Production.jsonof topic 13 switches the console log to thesystemdformat, and the development-only endpoints (/openapi,/scalar) are not mapped.Environment=ASPNETCORE_URLS=http://127.0.0.1:5000makes Kestrel listen on the loopback address only. Nothing outside the machine can reach the API directly; every request comes through Nginx.Restart=always,RestartSec=5— whenever the process ends, for whatever reason, systemd starts it again after five seconds.KillSignal=SIGINT,TimeoutStopSec=30— onstoporrestartsystemd sendsSIGINT, the signal of Ctrl+C, to which the .NET host reacts with a graceful shutdown: running requests finish, SignalR connections are closed, hosted services stop. Only if the process is still alive after 30 seconds is it killed.SyslogIdentifier=chatex-api— the name in front of every line in the journal.
[Install]
WantedBy=multi-user.target — systemctl enable links the unit into the normal boot target, so the API starts at every boot.
Step 3. The sandbox, directive by directive
Why. Running as chatex already limits the damage of a security hole. The hardening directives go much further. Each one closes a door that the API never uses, so they cost nothing at run time, and together they leave an attacker who takes over the process in a very small box: they can talk to the network and to the database, read the application and write uploads — and almost nothing else. Table 23.2 explains each directive.
Table 23.2. Hardening directives of chatex-api.service
| Directive | Effect for the process |
|---|---|
NoNewPrivileges=true | it and its children can never gain privileges, for example through set-user-id programs like sudo |
PrivateTmp=true | its own empty /tmp and /var/tmp, invisible to other programs and deleted when it stops |
PrivateDevices=true | only pseudo devices such as /dev/null and /dev/urandom; no disks, no terminals |
ProtectHome=true | /home, /root and /run/user appear empty |
ProtectSystem=strict | the whole file system is read-only, including /usr, /etc, /opt and /var |
ReadWritePaths=/var/lib/chatex | the single exception: the data folder stays writable |
ProtectKernelTunables=true | kernel settings in /proc/sys and /sys are read-only |
ProtectKernelModules=true | no kernel modules can be loaded |
ProtectControlGroups=true | the control groups, systemd's resource limits, are read-only |
RestrictSUIDSGID=true | it cannot create set-user-id or set-group-id files |
LockPersonality=true | it cannot switch the execution domain, an old trick to change kernel behaviour |
ProtectClock=true | it cannot change the system clock (which would break token expiry and TLS) |
RestrictNamespaces=true | it cannot create Linux namespaces, the building blocks of containers |
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK | sockets only for local Unix sockets, IPv4, IPv6 and netlink, which .NET uses to read the network interfaces |
CapabilityBoundingSet= | the empty list removes all Linux capabilities; even a process that became root could not bind low ports, change file owners or load modules |
UMask=0027 | new files are readable by the group and invisible to others (rw-r-----) |
The comment at the top of the block sums it up: the release is read-only, and only /var/lib/chatex is writable. The release folders are owned by root anyway (topic 24), so even without ProtectSystem the service could not modify its own code — the two protections back each other up.
How these directives work
None of this is magic in systemd. Before it starts the program, systemd creates a private mount namespace in which the file system is remounted read-only, empty or private as configured, drops the capabilities, sets the "no new privileges" flag, and installs a seccomp filter in the kernel that refuses the forbidden system calls. The program cannot undo any of it. The full list of options is in the manual page systemd.exec.
Step 4. Installing and checking the service
Do. Commit the unit and push it:
powershell
git add Deploypowershell
git commit -m "Add the systemd unit of the API"powershell
git pushOn the server, fetch it, install it, make systemd read it, and enable it for every boot:
bash
git pullbash
cp Deploy/systemd/chatex-api.service /etc/systemd/system/ && systemctl daemon-reload && systemctl enable chatex-apiCreated symlink '/etc/systemd/system/multi-user.target.wants/chatex-api.service' → '/etc/systemd/system/chatex-api.service'./etc/systemd/system/is the folder for units written by the administrator; units of packages live under/usr/lib/systemd/system/.daemon-reloadmakes systemd read the unit files again. It is needed after every change of a unit file.enablecreates the link ofWantedBy— the line above. It does not start the service, and it must not: there is no release yet, so/opt/chatex/currentdoes not exist. The service starts on the first deployment.
Check
bash
systemctl status chatex-api○ chatex-api.service - Chatex API
Loaded: loaded (/etc/systemd/system/chatex-api.service; enabled; preset: enabled)
Active: inactive (dead)The unit is loaded and enabled, and it is not running yet. Let systemd check the file for errors:
bash
systemd-analyze verify /etc/systemd/system/chatex-api.serviceThe command prints nothing when the unit is valid. Finally, let systemd rate the sandbox:
bash
systemd-analyze security chatex-apiIt lists each protection with a tick or a cross and ends with an overall exposure from 0 (fully locked) to 10 (unprotected). For this unit the last line is
→ Overall exposure level for chatex-api.service: 3.5 OK :-)A service without any hardening scores about 9.6, UNSAFE.
Logs of the service
journalctl -u chatex-api shows the journal of the unit, -f follows new lines live, -n 50 shows the last 50. Right now it says -- No entries --; after the first deployment you will see the start-up messages of the API there.