# DEPLOY - q4.suby.pl

Quake 4 browser port based on OpenQ4. The public service contains GPL engine code,
SDK-licensed gameplay modules, deterministic self-authored CC0 minimal `pak0`/`pak1`
packs, and an approved CC0 q4web arena addon. For this deployment, the operator has
confirmed authorization to deliver the required retail data. The config-gated gateway
serves only `pak001.pk4` through `pak022.pk4` plus `zpak_english.pk4` through
SHA-addressed URLs; the bytes remain outside Git, the public tree, generated content,
and the source offer.

| Item | Value |
|---|---|
| Public URL | `https://q4.suby.pl` |
| Direct URL | `https://q4.z6.mm.mk:11443` |
| Gateway | `suby:3280`, `q4.service` |
| Authoritative game | native OpenQ4 UDP `127.0.0.1:28004` |
| Working directory | `/srv/q4` |
| Health | `/healthz` |
| Status | `/api/status` |
| Retail catalog | `/retail/manifest.json`, exact 23-file allowlist |
| Replays | `/api/replays`, newest 300 finalized `.q4sd` files |
| Replay UDP pool | `127.0.0.1:28040-28047`, one isolated process per active replay |

## Normal operation

```bash
curl -s http://127.0.0.1:3280/healthz | python3 -m json.tool
curl -s http://127.0.0.1:3280/api/status | python3 -m json.tool
systemctl status q4
journalctl -u q4 -f

# The only permitted restart path. Exit 2 means it deliberately skipped.
cd /srv/q4 && tools/safe_restart.sh
```

The restart drains clients for up to ten minutes by default, then waits up to 120
seconds for `"status":"ok"`, `"dedicated":true`, and `"retail":true`. The dedicated
flag means OpenQ4 has loaded the authoritative map, not merely spawned; the retail
flag means the exact deployment catalog was validated and enabled. Override the post-restart
window with `Q4_RESTART_READY_TIMEOUT_SECONDS`; drain and polling controls are
`Q4_RESTART_DRAIN_POLLS`, `Q4_RESTART_DRAIN_INTERVAL_SECONDS`, and
`Q4_RESTART_READY_INTERVAL_SECONDS`.

Files under `public/` are served live. Changes to Node code, engine artifacts, native
binaries, maps used by the dedicated server, or the systemd environment require a
drained safe restart. Never run `systemctl restart q4` directly while the service is
already live.

Retail delivery defaults to disabled. Production enables it explicitly with
`Q4_RETAIL_DELIVERY=operator-authorized`, points `Q4_RETAIL_DIR` at the ignored
operator data and `Q4_RETAIL_MANIFEST` at the ignored staged manifest. Any other
non-disabled delivery mode is rejected. `Q4_RETAIL_MAX_CONCURRENT` and
`Q4_RETAIL_MAX_PER_IP` bound concurrent downloads; neither setting widens the exact
23-file allowlist.

`POST /api/replays/<id>/watch` allocates or reuses a replay process and returns its
`port` and same-origin `connectUrl`. `Q4_REPLAY_PORTS` is a comma-separated pool that
must not overlap `Q4_GAME_PORTS`; production uses `28040-28047`.
`Q4_REPLAY_PLAYBACK=0` disables allocation without hiding downloads. Optional lifecycle
knobs are `Q4_REPLAY_MAX_VIEWERS`, `Q4_REPLAY_START_TIMEOUT_MS`,
`Q4_REPLAY_END_GRACE_MS`, `Q4_REPLAY_IDLE_MS`, and `Q4_REPLAY_HARD_CAP_MS`.
Playback is frozen after process readiness until the first viewer reaches the in-game
state, so browser/WASM startup cannot consume a short replay before connection.

## Reproduce artifacts

Prerequisites include Node 22 or newer, GCC/G++ 13, Meson, Ninja, Emscripten as described by
`port/wasm/emscripten.ini`, SDL/OpenAL development headers, `innoextract`, `zip`, and
`rsync`.

```bash
cd /srv/q4
npm ci
tools/sync-upstream.sh

# Private operator data; output remains ignored and outside the public tree.
tools/import-retail.sh \
  '/mnt/smbgames/games/GOG/setup_quake_iv_1.4.3_lang_update_(63131).exe'

# Hash and validate the exact pak001-pak022 + zpak_english deployment catalog.
# The ignored manifest contains only metadata and SHA-addressed route paths.
node tools/stage-retail-manifest.mjs

tools/build-native.sh --clean --jobs 4
tools/build-wasm.sh --clean --jobs 4

# Build and ingest the tested CC0 arena, then regenerate the audited CC0 runtime.
python3 tools/build-map-source-pk4.py \
  --source mods/maps/q4web_arena --output /tmp/q4web_arena-source.pk4
node tools/ingest.mjs /tmp/q4web_arena-source.pk4 \
  --license CC0-1.0 --source self-authored --author Oivo --force --no-restart
node tools/stage-web-content.mjs
tools/stage-source-offer.sh
```

`engine/` is a tracked frozen deployment artifact. `runtime/` and `assets-src/` are
private operator state. The retail bytes are read from there by the dedicated exact
catalog route, but are never copied under `public/`, `content/`, `engine/`, or the
source archive. `content/assets/` is rebuilt only from audited `content-src/` inputs
and approved addons; `mods/dist/` and the generated public source archive are generated
and never committed.

## Pre-deploy verification

```bash
npm test
npm run test:zip-cache
python3 tools/audit-public-content.py \
  --public-dir content/assets/baseoq4 \
  --retail-dir assets-src/gog/q4base \
  --allowlist content-src/retail-duplicate-allowlist.json
node vendor/openq4/tools/tests/webgl2_shader_smoke.mjs
python3 tools/test-wasm-network.py
port/native/smoke-dedicated.sh
port/native/smoke-capacity.sh
port/native/smoke-server-demo-playback.sh
port/native/smoke-server-demo-join.sh

# Recreate the public source offer from the current checkout, then prove that
# its extracted and patched trees match this release's clean source verification.
source_verify="$(tools/build-wasm.sh --clean --verify-sources-only)"
printf '%s\n' "${source_verify}"
openq4_diff="$(sed -n 's/^openq4_patched_diff_sha256=//p' <<<"${source_verify}")"
gamelibs_diff="$(sed -n 's/^gamelibs_patched_diff_sha256=//p' <<<"${source_verify}")"
patch_manifest="$(sed -n 's/^patch_manifest_sha256=//p' <<<"${source_verify}")"
runtime_source="$(sed -n 's/^free_runtime_source_sha256=//p' <<<"${source_verify}")"
[[ "${openq4_diff}" =~ ^[0-9a-f]{64}$ \
  && "${gamelibs_diff}" =~ ^[0-9a-f]{64}$ \
  && "${patch_manifest}" =~ ^[0-9a-f]{64}$ \
  && "${runtime_source}" =~ ^[0-9a-f]{64}$ ]]
tools/stage-source-offer.sh
Q4_EXPECT_OPENQ4_DIFF_SHA256="${openq4_diff}" \
Q4_EXPECT_GAMELIBS_DIFF_SHA256="${gamelibs_diff}" \
Q4_EXPECT_PATCH_MANIFEST_SHA256="${patch_manifest}" \
Q4_EXPECT_RUNTIME_SOURCE_SHA256="${runtime_source}" \
  tools/test-source-offer.sh

# Start the release locally with publication disabled. This also removes a
# watchdog cron left by an earlier installation before systemd starts q4.
sudo deploy/install.sh --start-local
for attempt in $(seq 1 60); do
  health=$(curl -fsS http://127.0.0.1:3280/healthz 2>/dev/null || true)
  grep -q '"status":"ok"' <<<"$health" \
    && grep -q '"dedicated":true' <<<"$health" \
    && grep -q '"retail":true' <<<"$health" \
    && break
  sleep 2
done
health=$(curl -fsS http://127.0.0.1:3280/healthz)
printf '%s\n' "$health" >&2
grep -q '"status":"ok"' <<<"$health"
grep -q '"dedicated":true' <<<"$health"
grep -q '"retail":true' <<<"$health"

Q4_E2E_URL=http://127.0.0.1:3280 npm run test:e2e:shell
Q4_E2E_URL=http://127.0.0.1:3280 npm run test:e2e:full

# Prove that schema-4 evidence covers every required phase and is bound to the
# exact engine, content, and retail manifests that are about to be published.
node tools/verify-e2e-evidence.mjs

# This is the only publication step. It repeats the evidence check, rechecks
# local readiness, conditionally restores the route, and then enables its cron.
sudo deploy/install.sh --publish
```

The release gate additionally requires the full Playwright test to fetch and verify
the enabled server retail catalog into a clean browser OPFS profile before checking a
rendered campaign map, two simultaneous WebAssembly MP clients, bot presence, voice
peer connection, a replay viewer followed by a second late-joining viewer, and
desktop/mobile canvas pixel checks. Save evidence in `.selfcheck/`. Never run
`--publish` until that command has completed successfully.

`--publish` first validates the source archive checksum and its embedded
`SOURCE_MANIFEST.json` against the current frozen `engine/engine-manifest.json`,
`engine/build-info.txt`, patch/runtime identifiers, and every artifact covered by
`engine/SHA256SUMS`. It then runs `tools/verify-e2e-evidence.mjs`, still before its
first health, Caddy, or cron action. The evidence verifier requires schema 4, a passed
run which verified the deployed release, and campaign, multiplayer, replay, and mobile
evidence whose individual checks carry the exact current engine/content/retail release
identity. Required phases may be retained across separate E2E invocations when the
artifact identity is unchanged; the most recent run does not need to request all four
phases at once.

The isolated native capacity smoke holds eight bots for 30 seconds, connects two named
native clients, then requires the same eight-bot floor for another 30 seconds. Its four
delimited `serverInfo` snapshots and process logs remain in
`runtime/native-capacity-smoke/`.

Do not satisfy the source-offer gate with an archive left by an earlier build. The
pre-deploy sequence above regenerates it only after obtaining the current clean
source identity, then requires the extracted archive to reproduce both patched-tree
hashes plus the patch-manifest and free-runtime hashes. Publication independently
rejects an archive whose embedded release identity does not match the frozen engine.

`tools/stage-web-content.mjs` runs the same audit before publishing. It compares every
public PK4 entry with the private retail corpus by normalized path and content SHA-256;
exact byte duplicates fail whether they keep the same path or use a new name. The committed allowlist is
intentionally empty, and any future exception requires explicit license evidence.

## First installation

Stage and verify artifacts first. The installer creates the locked system identity
`q4`, transfers ownership only for the three runtime write locations, installs the
unit, and reloads systemd. Its default mode intentionally changes neither service nor
publication state:

```bash
cd /srv/q4
sudo deploy/install.sh
systemd-analyze verify /etc/systemd/system/q4.service

# Start a local-only release target before browser E2E. This removes an existing
# /etc/cron.d/q4-caddy-watchdog before enabling/starting q4.service.
sudo deploy/install.sh --start-local
for attempt in $(seq 1 60); do
  health=$(curl -fsS http://127.0.0.1:3280/healthz 2>/dev/null || true)
  grep -q '"status":"ok"' <<<"$health" \
    && grep -q '"dedicated":true' <<<"$health" \
    && grep -q '"retail":true' <<<"$health" \
    && break
  sleep 2
done
health=$(curl -fsS http://127.0.0.1:3280/healthz)
printf '%s\n' "$health" >&2
grep -q '"status":"ok"' <<<"$health"
grep -q '"dedicated":true' <<<"$health"
grep -q '"retail":true' <<<"$health"

# Run the complete pre-deploy suite above, including full Playwright E2E. Publish
# only after it passes; the installer rechecks local readiness and delegates route
# creation to the idempotent watchdog before enabling the two-minute cron.
sudo deploy/install.sh --publish
```

`--start-local` and `--publish` are idempotent. The former always removes the cron
before `systemctl enable --now q4`; the latter refuses to install it unless the
release-bound E2E evidence passes and local health reports `status=ok`,
`dedicated=true`, and `retail=true`. Publication never performs a
raw unconditional Caddy `POST`: `tools/caddy_route_watchdog.sh` first exits when the
host route already exists. Re-running `--publish` therefore checks the existing route
and refreshes the same cron file without duplicating it.

The service runs as the non-login system user/group `q4` with an empty capability
set, `NoNewPrivileges`, private devices/tmp, and `ProtectSystem=strict`. Only
`replays/`, `runtime/native/home/`, and `runtime/replay-instances/` are writable in
its mount namespace. The exact retail catalog is read-only and reachable only through
the manifest and SHA-addressed allowlist routes; retail PK4s, native
binaries/GameLibs, public files, content, and engine artifacts remain root-owned and
read-only. After restaging native or retail files, verify they are
world-readable/traversable for `q4`; do not recursively chown the checkout.

The `q4.suby.pl` Caddy route proxies HTTPS/WSS to `10.0.0.150:3280`. The direct
`q4.z6.mm.mk:11443` route is generated automatically from `/srv/gry/server.js` after
the portal registry contains `{ id: 'q4', port: 3280 }`. Client URLs and sockets are
same-origin, so no special direct-domain engine build is needed.

After the gateway is publicly healthy, add the q4 entry and tile to `/srv/gry`, then
restart the portal only through `/srv/gry/tools/safe_restart.sh`.

## Recovery

1. Restore the tracked `/srv/q4` tree and run `npm ci`.
2. Recreate pinned vendor trees with `tools/sync-upstream.sh`.
3. Re-extract the operator data with `tools/import-retail.sh`, then regenerate the
   exact deployment catalog with `node tools/stage-retail-manifest.mjs`.
4. Rebuild native and WebAssembly artifacts, ingest the CC0 arena, stage the audited
   self-authored CC0 content, and stage the source offer using the commands above.
5. Run `deploy/install.sh`, then `deploy/install.sh --start-local`; verify local health
   reports `status=ok`, `dedicated=true`, and `retail=true`.
6. Run every pre-deploy and full-browser check, then run
   `deploy/install.sh --publish`. Restore the portal tile last.

Replay recordings are runtime evidence, not required application state. If retained,
restore only complete `.q4sd` plus matching metadata; discard `.part` files. Browser
saves and verified retail PK4s live in each user's origin-private OPFS. A missing
server-catalog revision is downloaded and verified again on the next game launch;
manual-import revisions remain local to that browser.

## Teardown

```bash
systemctl disable --now q4
rm -f /etc/cron.d/q4-caddy-watchdog
# Find the q4.suby.pl route index in the Caddy admin JSON, then DELETE that index.
```

Do not delete `assets-src/gog` unless the operator is prepared to re-extract the
licensed installer. Never copy it into a backup intended for public distribution.
