English
Activating a release
Step 1. What "activating" must guarantee
Why. The release arrives in /opt/chatex/incoming/<sha>. Turning it into the running version is the most delicate moment of the whole pipeline, and it has to be safe even when something goes wrong:
- the users must never see a half-copied or mixed version;
- the database must be migrated before the new code that needs the new schema starts;
- a version that does not start must not stay live;
- going back to an older version must be one command;
- the disk must not fill up with old releases.
Chatex solves this with the folder layout of topic 23 — complete releases side by side and the symbolic link current — and one Bash script that runs on the server as root, chatex-activate. Fig. 24.2 shows what it does.
Fig. 24.2. Activating a release with rollback
Step 2. The script
Do. On your computer, create the script in the folder Deploy:
bash
#!/usr/bin/env bash
# Activates a Chatex release on the production server.
#
# Install once as root (see Docs/Deployment.md):
# install -o root -g root -m 755 Deploy/activate-release.sh /usr/local/bin/chatex-activate
# The deploy workflow connects as root (Docs/Deployment.md) and calls it through sudo, which also works for a
# dedicated deploy user allowed only this command in sudoers. Keep it root-owned and outside anything that user
# can write, otherwise sudo on it would grant root.
#
# Usage: chatex-activate <git sha>
# - If /opt/chatex/incoming/<sha> exists (uploaded by the deploy workflow), it becomes
# /opt/chatex/releases/<sha>.
# - Otherwise /opt/chatex/releases/<sha> must already exist: this is how you roll back.
# Then it runs the database migrations, switches /opt/chatex/current, restarts the API and checks
# /health. If the API does not come up, it switches back to the previous release.
set -euo pipefail
readonly ROOT=/opt/chatex
readonly INCOMING="$ROOT/incoming"
readonly RELEASES="$ROOT/releases"
readonly CURRENT="$ROOT/current"
readonly ENV_FILE=/etc/chatex/chatex.env
readonly SERVICE=chatex-api
readonly HEALTH_URL=http://127.0.0.1:5000/health
readonly KEEP_RELEASES=5
log() { printf '[chatex-activate] %s\n' "$*"; }
fail() { log "error: $*" >&2; exit 1; }
[[ $EUID -eq 0 ]] || fail "must run as root"
[[ $# -eq 1 ]] || fail "usage: chatex-activate <git sha>"
readonly SHA="$1"
[[ $SHA =~ ^[0-9a-f]{7,40}$ ]] || fail "not a git sha: $SHA"
readonly SRC="$INCOMING/$SHA"
readonly DST="$RELEASES/$SHA"
PREVIOUS="$(readlink -f "$CURRENT" 2>/dev/null || true)"
readonly PREVIOUS
# 1. Move the upload into releases/ and make it read-only for the service user.
if [[ -d $SRC ]]; then
if [[ $PREVIOUS == "$DST" ]]; then
# A re-run of the same commit: keep the running files, just restart below.
rm -rf "$SRC"
log "$SHA is already installed and running"
else
rm -rf "$DST"
mv "$SRC" "$DST"
log "installed $DST"
fi
elif [[ -d $DST ]]; then
log "no upload for $SHA, re-activating the existing release"
else
fail "neither $SRC nor $DST exists"
fi
chown -R root:chatex "$DST"
chmod -R u=rwX,go=rX "$DST"
chmod 755 "$DST/efbundle"
[[ -f $DST/api/Chatex.Api.dll ]] || fail "$DST/api/Chatex.Api.dll is missing"
[[ -f $DST/web/index.html ]] || fail "$DST/web/index.html is missing"
# 2. Migrate. Migrations run while the old release still serves traffic, so they must stay
# compatible with it. Rolling back does not undo migrations.
# The env file uses systemd syntax, so read the value instead of sourcing it (';' would split).
CONNECTION="$(grep -m1 '^ConnectionStrings__Chatex=' "$ENV_FILE" | cut -d= -f2- || true)"
CONNECTION="${CONNECTION#\"}"
CONNECTION="${CONNECTION%\"}"
[[ -n $CONNECTION ]] || fail "ConnectionStrings__Chatex is not set in $ENV_FILE"
log "applying migrations"
"$DST/efbundle" --connection "$CONNECTION"
# 3. Switch the symlink atomically and restart.
switch_to() {
ln -sfn "$1" "$CURRENT.tmp"
mv -T "$CURRENT.tmp" "$CURRENT"
systemctl restart "$SERVICE"
}
healthy() {
local _
for _ in $(seq 1 30); do
if curl --fail --silent --max-time 2 "$HEALTH_URL" > /dev/null; then
return 0
fi
sleep 1
done
return 1
}
log "switching to $SHA"
switch_to "$DST"
if ! healthy; then
if [[ -n $PREVIOUS && -d $PREVIOUS && $PREVIOUS != "$DST" ]]; then
log "health check failed, rolling back to $(basename "$PREVIOUS")"
switch_to "$PREVIOUS"
fi
journalctl -u "$SERVICE" -n 50 --no-pager >&2 || true
fail "$SHA did not pass the health check"
fi
log "$SHA is live"
# 4. Clean up: leftover uploads and all but the newest releases (never the running one).
find "$INCOMING" -mindepth 1 -maxdepth 1 -type d -mmin +60 -exec rm -rf {} +
# Release folder names are hex SHAs, so parsing ls output is safe here.
# shellcheck disable=SC2012
ls -1dt "$RELEASES"/*/ | tail -n +$((KEEP_RELEASES + 1)) | while read -r old; do
old="${old%/}"
if [[ $old != "$DST" ]]; then
rm -rf "$old"
log "removed old release $(basename "$old")"
fi
doneThe header documents installation and usage. The parts of the script follow.
Strict mode, constants and the argument
set -euo pipefail— the "strict mode" of Bash:-estops the script at the first failing command,-utreats an unset variable as an error instead of an empty string,-o pipefailmakes a pipeline fail if any part of it fails, not only the last. Without it, a failedmvwould be followed by a cheerful restart of a release that is not there.readonlyconstants name every path, the service, the health URL and the number of releases to keep in one place.logandfailprint with the prefix[chatex-activate];failwrites to standard error and exits with code 1, which turns the workflow step red.[[ $EUID -eq 0 ]]— the script changes owners, restarts services and deletes folders; it refuses to run as anybody butroot.[[ $SHA =~ ^[0-9a-f]{7,40}$ ]]— the argument must look like a commit hash: 7 to 40 lower-case hexadecimal characters. This is a security check, not a formality: the value becomes part of paths that are deleted withrm -rf. An argument like../../etcnever gets that far.PREVIOUSremembers wherecurrentpoints now, as the full path of the release, for the rollback.|| truekeepsset -efrom stopping the script when there is no link yet.
1. Moving the upload into releases/
There are four cases:
- An upload exists, and its release is already running — a re-run of the same commit, for example with Run workflow. The running files are kept, the upload is deleted, and the script goes on to migrate and restart.
- An upload exists — the normal case. An older folder of the same name is removed, and the upload is moved into
releases/.mvwithin one file system is a rename: instant, never half done. - No upload, but the release exists — a rollback by hand:
chatex-activate <older sha>re-activates a kept release. - Neither — an error.
Then the release is made read-only for the service: chown -R root:chatex gives it to root with the group chatex, and chmod -R u=rwX,go=rX lets root write and everybody else only read (the capital X adds the execute bit to folders, which need it to be entered, but not to ordinary files). efbundle gets 755 back, because it must be executable and the upload arrives without execute bits. Finally two sanity checks: a release without api/Chatex.Api.dll or web/index.html is refused before anything changes.
2. Migrating the database
- The connection string comes from
/etc/chatex/chatex.env. The script does notsourcethe file: it is written in systemd syntax, not shell syntax, and the;inside the connection string would split it into several shell commands.grep -m1takes the first line with the key,cut -d= -f2-everything after the first=, and the two parameter expansions remove optional quotes. "$DST/efbundle" --connection "$CONNECTION"applies the migrations that the database does not have yet and prints them. On the first deployment it creates the whole schema; later it usually printsNo migrations were applied. The database is already up to date.
The migrations run while the old release still serves users. A migration must therefore stay compatible with the code before it: add a column as nullable or with a default, rename in two releases (add the new, copy, then drop the old later). This is the expand–contract pattern. The reverse is also true, and the script's comment says it plainly: rolling back does not undo migrations. The previous release must be able to run on the newer schema.
3. Switching and checking the health
switch_tocreates a new link under a temporary name,current.tmp, and renames it overcurrentwithmv -T. A rename is atomic in Linux: at every momentcurrentpoints either to the old or to the new release, never to nothing. (ln -sfndirectly oncurrentwould delete and recreate the link — two operations with a gap between them.) Then it restarts the service, and systemd startscurrent/api— the new code.healthyaskshttp://127.0.0.1:5000/healthup to 30 times, once a second.curl --failtreats503 Unhealthyas a failure,--max-time 2keeps a hanging request from blocking the loop. The API answersHealthyonly when it has started and can reach the database (topic 5), so the check covers wrong settings, a missing library and a broken connection string alike. The request goes directly to Kestrel with the host127.0.0.1, which is whyAllowedHostscontains it.- If the check fails, the script switches back to
PREVIOUSand restarts the old version — but only if there is a previous release and it is not the same folder. Then it prints the last 50 lines of the service's journal, which usually contain the reason, and fails. The new release stays inreleases/for inspection.
4. Cleaning up
find "$INCOMING" … -mmin +60 -exec rm -rf {} +deletes uploads older than an hour — leftovers of deployments that were interrupted between upload and activation.ls -1dt "$RELEASES"/*/ | tail -n +$((KEEP_RELEASES + 1))lists the releases newest first and skips the first five; every older one is deleted — except the active release, which is never removed. Five releases are five possible rollbacks.
A failed first deployment
On the very first deployment current does not exist yet, but readlink -f still prints the path it would have, /opt/chatex/current. If that first release fails its health check, the script reports health check failed, rolling back to current and leaves current as a link that points to itself. Nothing else is damaged: the API was not running before and is not running after. The next deployment of a working release replaces the link. (Exercise 2 of the checkpoint fixes this edge case.)
Step 3. Installing the script
Why. The script runs as root, so it must be impossible to change it without being root: it lives in /usr/local/bin, owned by root, outside every folder that a deployment writes to. That is also why the copy inside each release is never executed.
Do. Commit the script and push it:
powershell
git add Deploypowershell
git commit -m "Add the release activation script"powershell
git pushOn the server, in the folder chatex:
bash
git pullbash
cp Deploy/activate-release.sh /usr/local/bin/chatex-activate && chmod 755 /usr/local/bin/chatex-activate/usr/local/bin is on the PATH, so the command is now available as chatex-activate.
Check
Let Bash parse the script without running it:
bash
bash -n /usr/local/bin/chatex-activateNo output means no syntax errors. Then call it without and with a wrong argument:
bash
chatex-activate[chatex-activate] error: usage: chatex-activate <git sha>bash
chatex-activate 12345[chatex-activate] error: not a git sha: 12345ls -l /usr/local/bin/chatex-activate shows -rwxr-xr-x 1 root root.
How this was tested
The script, the unit and the Nginx site of this module were run in an Ubuntu 26.04 container with the packages of topic 23: a first deployment, a re-run of the same commit, a manual rollback, a release with a missing library — rolled back after 30 seconds with the error in the printed journal — and six deployments in a row, after which exactly five releases remained.