Coolify SaaS service user¶

Coolify Cloud (SaaS) deploys to our VMs by SSHing into a dedicated service account, not via the human admin user. This page documents that account on every GrinSystem VM that Coolify Cloud manages.

Applied to grintest-dev-01 by GRI-124.

Principles¶

• Service account, not human. coolify is an automated user owned by Coolify Cloud's control plane, not a person. No interactive login by default.
• Trusted automation, broad sudo. coolify has passwordless NOPASSWD:ALL sudo. Coolify Cloud needs this to install Docker, manage systemd services, drop config files under /etc, and upgrade itself. See § Why NOPASSWD:ALL for the trade-off — including why narrower allowlists don't work in practice.
• Emergency human access. The human admin's pubkey (e.g. filip's ~/.ssh/grintest_hetzner_dev.pub) is also installed in ~coolify/.ssh/authorized_keys so an operator can become coolify if the SaaS control plane is unreachable and we need to inspect or restart containers manually.
• Two-key audit trail. authorized_keys separates SaaS automation from human emergency access with comment suffixes (coolify-generated-ssh-key vs (filip emergency access)), so a glance at /var/log/auth.log or journalctl -u ssh makes the actor obvious.

What gets created¶

The docker group is created even before the Docker package is installed — Coolify's own installer (next ticket) will add Docker and the existing membership will pick it up.

Commands¶

Run as a sudo-capable user (filip) on the VM.

# 1. docker group (Docker not yet installed — group is created early so
#    the coolify user can join it before Docker is added)
sudo groupadd --system docker || true

# 2. coolify system user
sudo adduser --system --group --shell /bin/bash --home /home/coolify coolify
sudo usermod -aG docker coolify

# 3. Sudoers — NOPASSWD:ALL. See § Why NOPASSWD:ALL for rationale.
sudo tee /etc/sudoers.d/coolify <<'EOF'
# coolify service user: passwordless sudo for Coolify Cloud automation.
#
# Originally locked to docker commands only (GRI-124 spec) but Coolify
# Cloud's bootstrap and ongoing operation require broader access (apt,
# systemctl, file ops). Widened to NOPASSWD:ALL to match Coolify's
# documented expectations. The mitigating control is that the Coolify
# SaaS pubkey is the only remote credential — a compromise of Coolify
# Cloud itself implies root access regardless of sudoers granularity.
coolify ALL=(ALL) NOPASSWD:ALL
EOF
sudo chmod 440 /etc/sudoers.d/coolify
sudo visudo -c   # must say "parsed OK"

# 4. SSH directory
sudo install -d -o coolify -g coolify -m 700 /home/coolify/.ssh
sudo touch /home/coolify/.ssh/authorized_keys
sudo chown coolify:coolify /home/coolify/.ssh/authorized_keys
sudo chmod 600 /home/coolify/.ssh/authorized_keys

# 5. Install Coolify SaaS pubkey — get from Coolify Cloud dashboard's
#    "Add Server" step
echo 'ssh-ed25519 AAAA<...> coolify-generated-ssh-key' \
  | sudo tee -a /home/coolify/.ssh/authorized_keys

# 6. Install human admin pubkey for emergency access — pulled from
#    /home/<admin>/.ssh/authorized_keys so the bytes match exactly
sudo bash -c 'grep "^ssh-" /home/filip/.ssh/authorized_keys \
  | sed "s/$/ (filip emergency access)/" \
  >> /home/coolify/.ssh/authorized_keys'

sudo chown coolify:coolify /home/coolify/.ssh/authorized_keys
sudo chmod 600 /home/coolify/.ssh/authorized_keys

Verification¶

From a laptop with the human admin key loaded in ssh-agent:

# Connect as coolify with the emergency key
ssh -i ~/.ssh/grintest_hetzner_dev coolify@dev.grintest.pl

Inside the shell:

# Passwordless sudo works — no password prompt
sudo whoami
# expect: root

# Confirm coolify is in the docker group
groups
# expect: coolify docker

From the Coolify Cloud dashboard, "Validate Server" should turn green — Coolify connects via its SaaS-generated keypair, runs its diagnostic suite (installs Docker if absent, drops its agent under /data/coolify, verifies the network), and reports the server as healthy.

Why NOPASSWD:ALL¶

The original GRI-124 spec called for sudo locked to docker, docker-compose, and systemctl ... docker commands only. That spec turned out to be incompatible with Coolify Cloud in practice.

What broke¶

After installing the restricted sudoers, the Coolify dashboard's Validate Server action failed with:

sudo: I'm sorry coolify. I'm afraid I can't do that

That's the default sudo rejection message — Coolify's bootstrap was running a non-allowlisted command (likely apt-get install or systemctl enable).

Why narrower allowlists don't work¶

Coolify Cloud needs sudo not just for the initial Docker install but for ongoing operation:

• Auto-updating the Coolify agent
• Installing new system packages as features are enabled
• Managing systemd units (Coolify sometimes wraps non-Docker workloads in systemd, and uses systemctl enable/disable more broadly than the original allowlist permitted)
• Editing files under /etc/docker/daemon.json, /etc/systemd/system/*, /data/coolify/
• Coolify version upgrades that change which binaries are invoked

A narrower allowlist that handles all current commands would still break the next time Coolify changed its bootstrap script. The allowlist would become a constant maintenance burden of "what's it calling this week".

The trade-off we accepted¶

• Loss: NOPASSWD:ALL means the Coolify SaaS keypair is effectively a root credential. If Coolify Cloud is breached, our dev VM is compromised at root level.
• Gain: a working deployment platform that doesn't need hand-holding for every Coolify update.
• Mitigations:
  • The Coolify SaaS pubkey is the only remote credential with this access (plus the human admin emergency key, which we already trust at root level).
  • SSH is key-only, password auth disabled.
  • fail2ban rate-limits SSH brute force.
  • The Hetzner Cloud Firewall + host ufw limit network exposure to 22/80/443.
  • This is the dev VM, not prod. Prod gets its own ticket where we revisit the threat model — likely landing on the same answer but with extra controls (Tailscale-only SSH, separate Coolify project, network segmentation).
• Realistic perspective: a compromise of Coolify Cloud itself means the attacker can push containers to any of their customers' servers via the agent that's installed. Sudoers granularity is not the limiting factor in that scenario.

When narrower might be possible¶

The only realistic path back to a docker-only allowlist is Option C from the GRI-124 discussion: install Docker and the Coolify agent manually as the human admin, then remove sudo from coolify entirely (just keep docker group membership). This would work until Coolify's next agent update requires sudo, at which point you'd be re-running the install manually. Not a stable pattern for a dev VM.

If this ever moves to prod, revisit by reading Coolify's own threat model and operational requirements at that time — they may evolve.

Why the human key sits in ~coolify/.ssh¶

Strictly speaking, an emergency operator could sudo -u coolify -i from their own admin shell. We install the human pubkey in ~coolify/.ssh anyway because:

1. It works even if the local sudo rules are broken (e.g. a typo in /etc/sudoers made filip's passwordless sudo fail). The fallback is "SSH directly as coolify".
2. The audit trail is cleaner: the SSH login as coolify is logged with the human's key fingerprint and comment, so it's obvious who the actual operator was.
3. Coolify Cloud may at any time rotate its own SaaS key. If we lose the SaaS connection during that rotation, the human key keeps admin-level access to the coolify account available.

Codification path¶

Same staging order as the rest of the host baseline (see host-baseline.md § Codification roadmap):

1. cloud-init — create the coolify user, install the public keys, drop the sudoers file. The fastest win and the most repeatable step for new VMs.
2. Ansible — idempotent role applies the same on existing VMs, handles pubkey rotation (Coolify's SaaS key or human key change).
3. Terraform — not applicable here; this is OS-level config, not cloud-provider infrastructure.

Until those exist, the commands above are the source of truth — run them on each new VM that Coolify Cloud will manage.

Deviations from the original ticket¶

• Sudo scope. Ticket asked for "passwordless sudo for docker commands only". Actual implementation is NOPASSWD:ALL — see § Why NOPASSWD:ALL above for the full rationale.
• Docs location. The ticket asked for docs/runbooks/server-bootstrap.md. We placed the content at docs/ops/coolify-user.md because:
  • docs/ops/ already exists (from GRI-123) and is the established home for infra / hardening docs.
  • This page is specific to one service (Coolify), not a general bootstrap; it pairs cleanly with host-baseline.md.
  • Splitting docs across two top-level folders for similar concerns would be churn. If a docs/runbooks/ is needed later for incident playbooks, that's its own ticket.