Skip to content
CatalystCatalyst Docs

Node Issues

Prerequisites

Section titled “Prerequisites”
  • Admin access to the panel, so you can read node status, metrics, and logs.
  • Shell access on the node (root or a sudo-capable account) to inspect and restart catalyst-agent.

Node shows unreachable

Section titled “Node shows unreachable”

Running servers keep running — only management is down. Check in order:

  1. Agent process on the node: systemctl status catalyst-agent (or the OpenRC equivalent). Restart it and watch its logs: journalctl -u catalyst-agent -n 100 --no-pager.
  2. Node → panel path: DNS resolves the panel URL from the node? Outbound HTTPS reaches it? A proxy or firewall change breaks this silently.
  3. Panel side: is the panel itself healthy (panel issues)? An unreachable panel makes every node look dead.
  4. Credentials: rotated or revoked node API keys break check-in — re-register the key (registering).

Agent installed but never checks in

Section titled “Agent installed but never checks in”
  • Wrong panel URL (scheme/host/port) in the agent config — the most common cause.
  • Clock skew: large time differences break TLS and token validation; sync with NTP.
  • Version mismatch: an agent far newer/older than the panel may be rejected — keep them in step (updating a node).

Agent unit will not start (SELinux label, start limit)

Section titled “Agent unit will not start (SELinux label, start limit)”

If systemctl status catalyst-agent shows the unit failing within a second or restarting forever, check the exact exit reason in journalctl -u catalyst-agent -n 50 --no-pager:

  • status=203/EXEC with Permission denied — the binary carries the wrong SELinux label. Agents installed before v1.85.0 were moved into place from /tmp, which keeps the source label (user_tmp_t), and systemd may not execute that. Re-run the node install command from the panel — it runs deploy-agent.sh, which now installs with install(1) and calls restorecon (installing a node) — or fix it in place: sudo restorecon -F /opt/catalyst-agent/catalyst-agent && sudo systemctl restart catalyst-agent.
  • start request repeated too quickly — the unit hit its crash-loop bound (10 starts per 60 s) instead of restarting silently forever. Fix the underlying start failure above, then clear the counter with sudo systemctl reset-failed catalyst-agent && sudo systemctl start catalyst-agent.

A node running agent v1.85.0 or later bounds the loop this way; an older unit keeps retrying forever and buries the real error in the journal.

All servers on one node suffer

Section titled “All servers on one node suffer”

That is a node problem, not N server problems. Check node metrics first (disk full, memory exhausted), consider rolling back a recent OS/runtime/agent update (updating a node), and migrate critical servers to a healthy node while you investigate.

Servers fail to start on one node

Section titled “Servers fail to start on one node”

Creating and installing work, but every start fails on the same node. Check the agent log on the node for one of these:

  1. Cannot mount disk image for <server-uuid> ... Using the plain data directory; disk quota will not be enforced. — the node cannot create loop mounts (the underlying error contains not permitted, Permission denied, or EPERM). The agent deliberately continues with a plain data directory instead of refusing to start, so servers still run — but disk quota is not enforced on this node. If a later start also logs Migration directory already exists: <path>, remove the stale <data-dir>/migrate/<server-uuid> directory (default data dir /var/lib/catalyst) and start the server again.
  2. io.weight / cgroup IO error from runc — the io cgroup v2 controller is not delegated to the agent, so the runtime cannot apply the server’s IO weight. This is common inside an unprivileged container. Run the node on a host (or a privileged container) whose cgroup v2 lists io in /sys/fs/cgroup/cgroup.controllers. The agent omits the IO weight automatically when the controller is absent, so keep the agent up to date.
  3. A SERVER_NOT_STOPPED / “Server must be stopped” message is not a node fault — that check runs in the panel.

An unprivileged LXC container is the usual cause of 1 and 2. It can host a node, but expect no disk quota and verify the io controller is available before offering the node to users.

Backup restores fail with “Function not implemented”

Section titled “Backup restores fail with “Function not implemented””

On Fedora 44 every archive restore aborts with Function not implemented (ENOSYS) and the server returns to Stopped with that error. Fedora’s GNU tar issues openat2(2) when extracting, and systemd’s RestrictSUIDSGID=yes denies that syscall outright: classic seccomp cannot inspect the mode field inside struct open_how, so systemd blocks the whole call (systemd#43314, closed upstream as “fix in GNU tar”).

The agent unit therefore ships RestrictSUIDSGID=no. Every other sandbox directive stays on (SystemCallFilter=@system-service @mount, NoNewPrivileges=true, ProtectSystem=full, ProtectHome=true, RestrictNamespaces=mnt net uts ipc, the capability bounding set), and the archive path keeps its own protections: restores refuse archive-supplied ownership and permissions, refuse device entries and any member or link that would escape the server directory before extraction, and game servers run as unprivileged container users. Symbolic and hard links inside the server directory restore normally — see backups.

  1. Confirm the cause — on the node run systemd-run -p RestrictSUIDSGID=yes --wait --pipe tar -cf /dev/null -C /tmp .. On Fedora 44 it fails; the agent log shows the same error for the restore.
  2. Check the effective unit — systemctl show catalyst-agent -p RestrictSUIDSGID must print RestrictSUIDSGID=no. Drop-ins override the main unit, so also run grep -r RestrictSUIDSGID /etc/systemd/system/catalyst-agent.service.d/.
  3. Apply the fix — the agent repairs an upgraded-in-place node on its next start: it writes /etc/systemd/system/catalyst-agent.service.d/zz-catalyst-openat2-compat.conf and restarts once. To do it immediately, re-run the deploy script for the node, or set RestrictSUIDSGID=no in the unit and any drop-in, then systemctl daemon-reload && systemctl restart catalyst-agent.

Data to collect before asking for help

Section titled “Data to collect before asking for help”

Agent version (catalyst-agent --version), the agent log tail (journalctl -u catalyst-agent -n 100 --no-pager), node OS and free resources (df -h, free -m), and whether the panel URL is reachable from the node with curl. The diagnostics bundle collects all of this in one redacted file.

Section titled “Related”