Skip to content
CatalystCatalyst Docs

Disaster Recovery

What you must be able to rebuild

Section titled “What you must be able to rebuild”
  1. Panel database — users, servers, roles, audit logs: everything except the game files. It lives in the catalyst-postgres-data volume (Compose prefixes the name with the project name).
  2. Panel configuration — the compose directory the installer created (./catalyst-docker under the directory you ran it from) with its .env. The .env holds POSTGRES_PASSWORD, BETTER_AUTH_SECRET, API_KEY_SECRET, and BACKUP_CREDENTIALS_ENCRYPTION_KEY; losing them invalidates sessions, API keys, and stored backup credentials even if the database survives.
  3. Node agents — reproducible: reinstall from the panel with a fresh Deploy command (installing a node). Game data comes back from user backups (user backups).
  4. File volumes — catalyst-server-data, catalyst-backup-data, and catalyst-plugin-data on the panel host. Redis has no volume: it is an ephemeral cache and is rebuilt from the database automatically.

Back up the panel

Section titled “Back up the panel”

Schedule, retention, and the storage backend are covered in backups; this section is the command reference. Run these on the panel host, from the compose directory. POSTGRES_USER and POSTGRES_DB default to catalyst and catalyst_db; use your .env values, and read CATALYST_COMPOSE_DIR there if you have moved the stack.

Terminal window
cd ./catalyst-docker                # the directory install.sh created
docker exec catalyst-postgres pg_dump -U catalyst -d catalyst_db \
  > catalyst-panel-$(date +%F).sql
tar czf catalyst-config-$(date +%F).tgz \
  .env docker-compose.yml docker-compose.*.yml nginx caddy traefik

To capture the file volumes as well, archive each one with a throwaway container. Compose prefixes volume names with the project name (the compose directory name), so discover the real names first:

Terminal window
docker volume ls --format '{{.Name}}' | grep -E '_catalyst-(server|backup|plugin)-data$' \
  | while read -r v; do
      docker run --rm -v "$v":/data -v "$PWD":/backup alpine \
        tar czf "/backup/${v}.tgz" -C /data .
    done

Ship the dump, the config archive, and the file archives off the host. Check that the dump is non-empty before you trust it.

Restore the panel

Section titled “Restore the panel”
  1. On the replacement host, install a fresh stack with the one-liner: curl -fsSL https://github.com/catalystctl/catalyst/releases/latest/download/install.sh | sudo bash (installing Catalyst). Stop it before importing: docker compose down — never down -v, which deletes the volumes.
  2. Put the saved .env back into the compose directory. Restoring the original secrets is what makes existing sessions, API keys, and backup credentials valid again.
  3. Start PostgreSQL only, then restore the dump into it:
    Terminal window
    docker compose up -d postgres
    cat catalyst-panel-<date>.sql | docker exec -i catalyst-postgres \
      psql -U catalyst -d catalyst_db
  4. Restore the file volumes, if you captured them. The volume names must match the ones Compose expects, which is why the compose directory keeps the same name:
    Terminal window
    shopt -s nullglob
    for f in catalyst-*_catalyst-{server,backup,plugin}-data.tgz; do
      v="${f%.tgz}"
      docker run --rm -v "$v":/data -v "$PWD":/backup alpine \
        sh -c "rm -rf /data/* && tar xzf /backup/$f -C /data"
    done
  5. Start the rest of the stack: docker compose up -d.
  6. Log in, open Admin → Nodes, and confirm each node reconnects. Nodes that were wiped or rebuilt need a fresh agent install; their game data comes from backups on the node or in the backup volume.

bash update.sh --restore <backup-dir> restores the compose files from the backup update.sh takes before it rewrites them — useful if an update is what broke the stack.

Restore failures that are not the archive’s fault

Section titled “Restore failures that are not the archive’s fault”

A restore that dies partway is usually the node, not the backup. Two known causes, both fixed in current agent releases — keep nodes updated before drilling:

  • Function not implemented (ENOSYS) on Fedora 44. Fedora’s GNU tar uses openat2(2) when extracting, and systemd’s RestrictSUIDSGID=yes blocks that syscall. The agent unit now ships RestrictSUIDSGID=no with the other sandbox directives intact, and a node upgraded in place repairs itself on the next agent start. See backup restores fail with Function not implemented.
  • Symbolic and hard links in the archive. Older agents refused them. Current agents restore links inside the server directory as they were and still refuse archive-supplied ownership, permissions, device entries, and any member or link that would escape the server directory.

The recovery drill (quarterly)

Section titled “The recovery drill (quarterly)”

If you have never restored, you do not have recovery. Take a fresh dump, config archive, and file archive; restore them into a scratch host as above; log in, open servers, list backups, check the audit log; and time the whole thing — that duration is your real RTO.

During a real incident: declare who is in charge and where status is posted, restore in dependency order (database → panel → nodes → verify servers), and keep the recovery instructions somewhere other than the host they recover.

Section titled “Related”