operational deployment

.omo/run-continuation/ses_f6341e990ffeuDX0xnVFWQQ8fS.json

@@ -0,0 +1,10 @@
+{
+  "sessionID": "ses_f6341e990ffeuDX0xnVFWQQ8fS",
+  "updatedAt": "2026-09-13T22:17:31.684Z",
+  "sources": {
+    "background-task": {
+      "state": "idle",
+      "updatedAt": "2026-09-13T22:17:31.684Z"
+    }
+  }
+}

.omo/run-continuation/ses_f63e23ebcffe7Q6sJKCl9W8sFt.json

@@ -1,10 +1,10 @@
 {
   "sessionID": "ses_f63e23ebcffe7Q6sJKCl9W8sFt",
-  "updatedAt": "2026-09-13T19:05:07.347Z",
+  "updatedAt": "2026-09-13T21:32:58.318Z",
   "sources": {
     "background-task": {
       "state": "idle",
-      "updatedAt": "2026-09-13T19:05:07.347Z"
+      "updatedAt": "2026-09-13T21:32:58.318Z"
     }
   }
 }

README.md

@@ -1,2 +1,175 @@
-# stoat
+# Stoat — Fedora 44 self-hosted chat deployment
 
+Ansible automation that **hardens a Fedora 44 server**, installs **Podman**, and deploys a self-hosted **[Stoat](https://stoat.chat)** instance (open-source community chat, formerly Revolt) using the official [`stoatchat/self-hosted`](https://github.com/stoatchat/self-hosted) stack.
+
+**Target host:** `stoat-demo`
+
+---
+
+## Project layout
+
+```
+├── site.yml                     # main playbook — applies the role
+├── inventory.ini                # server inventory (stoat-demo)
+├── group_vars/all/vars.yml      # ★ all deployment variables live here
+└── roles/stoat/
+    ├── tasks/
+    │   ├── main.yml             # imports, in order
+    │   ├── system.yml           # dnf full update + base packages
+    │   ├── harden.yml           # firewalld, sshd, fail2ban, dnf5-automatic, logrotate
+    │   ├── podman.yml           # podman, compose, SELinux container policy
+    │   └── stoat.yml            # clone repo → generate config → systemd unit → start
+    ├── handlers/main.yml
+    └── templates/
+        ├── jail.local.j2                    # fail2ban config
+        └── podman-compose-stoat.service.j2  # systemd unit for the stack
+```
+
+---
+
+## Before you run
+
+1. **Set the real domain** in `group_vars/all/vars.yml`:
+
+   ```yaml
+   stoat_domain: chat.yourdomain.com   # ← A/AAAA record must point at the server
+   ```
+
+   Let's Encrypt certificates are issued for this name, and every URL Stoat advertises (web client, API, WebSocket, LiveKit) uses it. Do **not** use the public `stoat.chat` domain for your own instance.
+
+2. **Set the SSH user** in `inventory.ini`:
+
+   ```ini
+   stoat-demo ansible_host=stoat-demo ansible_user=YOUR_SSH_USER
+   ```
+
+3. **SSH is not touched by this playbook** — the server keeps its SSH configuration exactly as you have it. Fail2Ban still protects the login (`3` failed attempts → `1 h` ban, tunable). Ansible connects with your SSH password via `-k`:
+
+   ```bash
+   ansible-playbook -i inventory.ini site.yml
+   ```
+
+---
+
+## What the playbook does
+
+| Stage | Actions |
+|---|---|
+| **System** | `dnf update` all packages; install `git`, `curl`, `openssl` |
+| **Firewall** | installs + starts firewalld (native/minimal Fedora may not ship it); default-deny; allow `ssh`, `http`, `https`, `7881/tcp`, `50000-50100/udp` (LiveKit voice) |
+| **Fail2Ban** | `sshd` jail (3 retries → 1 h ban, incremental for repeat offenders) |
+| **Auto-updates** | `dnf5-plugin-automatic`, security-only, `apply_updates = yes`, started timer |
+| **Logs** | logrotate rule for fail2ban logs (daily, 7 rotations, compressed) |
+| **Podman** | engine + `podman-compose` + `podman-docker`; runs as root (no unprivileged service user, no linger) |
+| **SELinux** | keeps mode enforcing; installs `container-selinux` (container policy) + `policycoreutils-python-utils` / `setools-console` (`semanage`, `audit2why`); registers the deployment tree as `container_file_t` so rootful containers can use the bind mounts |
+| **Stoat** | clone `stoatchat/self-hosted` to `/opt/stoat/self-hosted`; run `generate_config.sh <domain>` (answers: built-in Caddy, video enabled); start stack under `podman-compose-stoat.service` |
+
+---
+
+## Network / ports
+
+| Port | Purpose |
+|---|---|
+| 22/tcp | SSH |
+| 80/tcp, 443/tcp | Caddy reverse proxy (automatic Let's Encrypt TLS) |
+| 7881/tcp | LiveKit WebRTC signaling |
+| 50000–50100/udp | LiveKit voice/video media (cannot be proxied) |
+
+---
+
+## Variables reference (`group_vars/all/vars.yml`)
+
+| Variable | Default | Meaning |
+|---|---|---|
+| `stoat_domain` | `chat.example.com` | **Required** — public domain for this instance |
+| `stoat_install_dir` | `/opt/stoat/self-hosted` | where the stack lives on the server |
+| `stoat_version` | `main` | git ref of `self-hosted` repo |
+| `stoat_reverse_proxy` | `false` | `true` = move Caddy to `:8880`, bring your own front proxy/TLS |
+| `stoat_video` | `true` | enable camera/screen sharing (LiveKit) |
+| `selinux_mode` | `enforcing` | SELinux mode the role enforces (config + setenforce) |
+| `fail2ban_port` / `fail2ban_maxretry` / `fail2ban_bantime` | `22` / `3` / `3600` | sshd jail tuning |
+| `firewall_services` / `firewall_ports` | ssh,http,https / 7881, UDP range | firewalld allow rules |
+
+Override any variable per-run, without editing files:
+
+```bash
+ansible-playbook -i inventory.ini site.yml -k -e "stoat_domain=chat.example.org fail2ban_maxretry=5"
+```
+
+---
+
+## After deployment
+
+1. **First account = instance owner.** Open `https://<your-domain>` and register.
+2. **Make it invite-only** (recommended for a private instance): in `/opt/stoat/self-hosted/Revolt.toml` set `invite_only = true`, then restart the API service.
+3. **Back up `secrets.env`** — it holds the VAPID push keys, file-encryption key, and LiveKit credentials. Losing it means losing access to **all uploaded files** and breaking existing sessions. Copy it somewhere safe:
+   ```bash
+   scp stoat-demo:/opt/stoat/self-hosted/secrets.env ~/backups/
+   ```
+4. **Caddy terminates TLS itself** — do not put Cloudflare proxy (orange cloud) in front of the domain; ACME needs direct access to the server (grey cloud / direct DNS only).
+
+---
+
+## Day-to-day operations
+
+All commands run on the server as root:
+
+```bash
+# Stack status
+systemctl status podman-compose-stoat
+podman ps
+
+# Logs (per service: api / events / web / livekit / database …)
+podman logs -f api
+
+# Restart the whole stack
+systemctl restart podman-compose-stoat
+
+# Update Stoat to new images (run in /opt/stoat/self-hosted)
+podman-compose pull
+podman-compose up -d
+
+# Fail2Ban status
+sudo fail2ban-client status sshd
+
+# Open firewall UI changes
+sudo firewall-cmd --list-all
+```
+
+---
+
+## SELinux notes
+
+Fedora 44 ships SELinux **enforcing**; the role pins that mode. Containers run as root under the `container_t` domain, so the role registers `/opt/stoat/self-hosted` as **`container_file_t`** (via `semanage fcontext` + `restorecon`) — that's what lets the rootful containers read/write the bind-mounted `data/` volumes and config files without per-volume `:Z` flags, and the registration is persistent (survives `restorecon`).
+
+If a service misbehaves, check for denials before anything else:
+
+```bash
+sudo ausearch -m avc -ts recent | grep -i stoat    # recent denials
+# or find the exact rule suggestion:
+sudo grep "denied" /var/log/audit/audit.log | tail -20 | audit2why
+```
+
+Common knobs if a denial shows up (only set what the denial actually requires):
+
+```bash
+# containers connecting to non-default ports on the host
+sudo setsebool -P container_connect_any 1
+# container needs to manage cgroups (systemd inside a container)
+sudo setsebool -P container_manage_cgroup 1
+# re-apply file contexts if you add bind mounts outside the
+# deployment tree (the role only labels /opt/stoat/self-hosted):
+sudo semanage fcontext -a -t container_file_t '/opt/stoat/moredata(/.*)?'
+sudo restorecon -RF /opt/stoat/moredata
+```
+
+---
+
+## Caveats & notes
+
+- **First start pulls ~2 GB of images** and initializes MongoDB — allow a few minutes; the systemd unit tolerates up to 15 min (`TimeoutStartSec=900`).
+- **Podman, not Docker**: the upstream stack is docker-compose based; this deployment runs it with `podman-compose` as root. It supports the compose features this stack needs (`depends_on: condition: service_healthy`, bind mounts, port ranges).
+- **`generate_config.sh` is guarded** by `creates:` — it never overwrites existing `secrets.env`/`Revolt.toml` unintentionally. To regenerate config deliberately, run it manually with `--overwrite` (after backing up secrets).
+- **Security posture**: fail2ban, firewalld default-deny, security-only auto-updates, SELinux enforcing with `container_file_t` labels. SSH configuration is left exactly as you set it. (Aligned with the upstream guide's recommendations.)
+- **Data persistence**: everything persists in `/opt/stoat/self-hosted/data/` (mongodb, rabbitmq, minio, caddy) — keep it out of the git clone/reset.
+- System hardening + Stoat are one role run; to re-apply only config changes, re-running `site.yml` is idempotent.

group_vars/all/vars.yml

@@ -13,7 +13,7 @@
 # for this name, and every URL Stoat advertises uses it.
 # Example: stoat.example.com  (NOT the public stoat.chat domain)
 # ------------------------------------------------------------
-stoat_domain: chat.example.com
+stoat_domain: chat.fpov.space
 
 # ------------------------------------------------------------
 # Stoat application
@@ -33,20 +33,13 @@ stoat_reverse_proxy: false
 stoat_video: true
 
 # ------------------------------------------------------------
-# Podman (rootless)
+# Podman (runs as root)
 # ------------------------------------------------------------
-# Unprivileged system user that runs the rootless podman stack
-stoat_service_user: stoat
-
 podman_packages:
   - podman
   - podman-compose          # compose file runner (supports service_healthy conditions)
   - podman-docker           # `docker` CLI compatibility, backed by podman
 
-# Rootless containers cannot bind ports < 1024 by default.
-# Caddy needs 80/443, so lower the unprivileged port floor.
-podman_unprivileged_port_start: 80
-
 # ------------------------------------------------------------
 # Hardening - Fail2Ban
 # ------------------------------------------------------------
@@ -58,13 +51,23 @@ fail2ban_bantime: 3600      # ban length in seconds (1 hour)
 # Hardening - SSH
 # ------------------------------------------------------------
 sshd_permit_root_login: no
-sshd_password_authentication: no
+# Keep password login enabled on this server. Fail2Ban (see below)
+# still guards against brute-force attempts.
+sshd_password_authentication: yes
 
 # Enable ONLY if a TOTP / WebUI-2FA PAM module is already
 # configured in /etc/pam.d/sshd - otherwise you can lock yourself
 # out. Leave false unless you know what you are doing.
 sshd_mfa_required: false
 
+# ------------------------------------------------------------
+# Hardening - SELinux
+# ------------------------------------------------------------
+# Native Fedora 44 ships SELinux enforcing; the role keeps it in
+# this mode and installs container-selinux + policycoreutils
+# (semanage / audit2why) for diagnosing container denials.
+selinux_mode: enforcing
+
 # ------------------------------------------------------------
 # Hardening - firewall (firewalld, default deny)
 # ------------------------------------------------------------

inventory.ini

@@ -1,15 +1,16 @@
 # --------------------------------------------------------------
 # Ansible inventory - servers to harden
-# Usage: ansible-playbook -i inventory.ini harden_fedora44.yml
+# Usage: ansible-playbook -i inventory.ini site.yml -k
 # --------------------------------------------------------------
 
 [servers]
-stoat-demo ansible_host=stoat-demo ansible_user=YOUR_SSH_USER
+# Replace YOUR_SSH_PASSWORD with the real password. Keep the file
+# out of git / chmod 600 it, or encrypt with ansible-vault:
+#   ansible-vault encrypt inventory.ini
+stoat-demo
 
 [servers:vars]
 # Elevate privileges (the playbook already sets become: true,
 # but this allows per-group override if needed)
 ansible_become=yes
-ansible_become_method=sudo
+ansible_become_method=sudo
-# Optional: path to your SSH private key
-# ansible_ssh_private_key_file=/home/you/.ssh/id_ed25519

roles/stoat/tasks/harden.yml

@@ -1,7 +1,14 @@
 ---
 # ------------------------------------------------------------
 # Firewall (firewalld) - default deny, allow only what is needed
+# Native/minimal Fedora installs may not ship firewalld, so
+# install it explicitly before starting it.
 # ------------------------------------------------------------
+- name: Install firewalld
+  ansible.builtin.dnf:
+    name: firewalld
+    state: present
+
 - name: Start and enable firewalld
   ansible.builtin.systemd:
     name: firewalld
@@ -34,29 +41,6 @@
   when: item.rc | int != 0
   notify: Reload firewalld
 
-# ------------------------------------------------------------
-# SSH hardening (Drop-in config, validated before restart)
-# ------------------------------------------------------------
-- name: Deploy sshd hardening drop-in
-  ansible.builtin.template:
-    src: sshd_hardening.conf.j2
-    dest: /etc/ssh/sshd_config.d/00-hardening.conf
-    owner: root
-    group: root
-    mode: "0644"
-  register: sshd_conf
-
-- name: Validate sshd configuration
-  ansible.builtin.command: sshd -t
-  changed_when: false
-  when: sshd_conf is changed
-
-- name: Restart sshd
-  ansible.builtin.service:
-    name: sshd
-    state: restarted
-  when: sshd_conf is changed
-
 # ------------------------------------------------------------
 # Fail2Ban - brute force protection
 # ------------------------------------------------------------
@@ -88,19 +72,24 @@
     name: dnf5-plugin-automatic
     state: present
 
+# ini_file with create: true handles both a missing config file
+# (package may not ship /etc/dnf/automatic.conf until a run) and
+# preserves any other sections the user may have customized.
 - name: Enable automatic installation of updates
-  ansible.builtin.lineinfile:
+  community.general.ini_file:
     path: /etc/dnf/automatic.conf
-    insertafter: "^\\[commands\\]"
+    section: commands
-    regexp: "^#?\\s*apply_updates\\s*="
+    option: apply_updates
-    line: "apply_updates = yes"
+    value: "yes"
+    create: true
 
 - name: Restrict automatic updates to security errata
-  ansible.builtin.lineinfile:
+  community.general.ini_file:
     path: /etc/dnf/automatic.conf
-    insertafter: "^\\[commands\\]"
+    section: commands
-    regexp: "^#?\\s*upgrade_type\\s*="
+    option: upgrade_type
-    line: "upgrade_type = security"
+    value: "security"
+    create: true
 
 - name: Enable and start dnf5-automatic.timer
   ansible.builtin.systemd:

roles/stoat/tasks/main.yml

@@ -2,7 +2,7 @@
 # stoat role - task order matters:
 #   system  -> update + baseline packages (git needed by stoat tasks)
 #   harden  -> firewall, ssh, fail2ban, automatic updates, logrotate
-#   podman  -> rootless podman + service user
+#   podman  -> podman + compose + SELinux container policy
 #   stoat   -> clone repo, generate config, systemd-managed compose stack
 - name: System update and base packages
   ansible.builtin.import_tasks: system.yml
@@ -10,7 +10,7 @@
 - name: Harden the host
   ansible.builtin.import_tasks: harden.yml
 
-- name: Install rootless podman
+- name: Install podman
   ansible.builtin.import_tasks: podman.yml
 
 - name: Deploy Stoat

roles/stoat/tasks/podman.yml

@@ -1,33 +1,38 @@
 ---
 # ------------------------------------------------------------
-# Podman - rootless engine, compose support, unprivileged user
+# Podman - engine, compose support (runs as root, no service user)
 # ------------------------------------------------------------
 - name: Install podman stack
   ansible.builtin.dnf:
     name: "{{ podman_packages }}"
     state: present
 
-# Rootless podman cannot bind host ports < 1024 without this.
+# ------------------------------------------------------------
-# Caddy (Stoat's TLS reverse proxy) needs 80/443.
+# SELinux - keep enforcing, ensure the container policy and
-- name: Allow rootless containers to bind privileged ports
+# diagnostic tools are present for resolving any denials.
-  ansible.builtin.sysctl:
+# ------------------------------------------------------------
-    name: net.ipv4.ip_unprivileged_port_start
+- name: Install SELinux container policy and diagnostics tools
-    value: "{{ podman_unprivileged_port_start }}"
+  ansible.builtin.dnf:
-    sysctl_file: /etc/sysctl.d/99-podman.conf
+    name:
+      - container-selinux
+      - policycoreutils-python-utils
+      - setools-console
     state: present
-    reload: true
 
-- name: Create dedicated service user for rootless podman
+- name: Check current SELinux mode
-  ansible.builtin.user:
+  ansible.builtin.command: getenforce
-    name: "{{ stoat_service_user }}"
+  register: selinux_current_mode
-    system: true
+  changed_when: false
-    shell: /bin/bash
-    create_home: true
-    comment: "Stoat rootless podman service user"
-  register: stoat_system_user
 
-# Keeps /run/user/<uid> alive and lets rootless containers start
+- name: Configure SELinux mode in /etc/selinux/config
-# on boot without an interactive login.
+  ansible.builtin.lineinfile:
-- name: Enable lingering for the service user
+    path: /etc/selinux/config
-  ansible.builtin.command: "loginctl enable-linger {{ stoat_service_user }}"
+    regexp: '^SELINUX='
+    line: 'SELINUX={{ selinux_mode }}'
+
+- name: Apply SELinux mode now
+  ansible.builtin.command: "setenforce {{ selinux_mode }}"
+  when:
+    - selinux_mode | lower == "enforcing"
+    - selinux_current_mode.stdout | lower != "enforcing"
   changed_when: false

roles/stoat/tasks/stoat.yml

@@ -1,35 +1,56 @@
 ---
 # ------------------------------------------------------------
-# Stoat deployment - rootless podman-compose, managed by systemd
+# Stoat deployment - podman-compose (as root), managed by systemd
 # ------------------------------------------------------------
+# The whole stack runs as root (play-level become): the repo and
+# config are generated as root and the systemd unit runs
+# podman-compose without dropping privileges.
+# The deployment dir may already be owned by another user from
+# earlier runs; register it so git-as-root accepts it.
+- name: Allow git to operate on the deployment directory
+  ansible.builtin.command:
+    cmd: "git config --global --add safe.directory {{ stoat_install_dir }}"
+  changed_when: false
+
 - name: Clone Stoat self-hosted repository
   ansible.builtin.git:
     repo: "{{ stoat_repo }}"
     dest: "{{ stoat_install_dir }}"
     version: "{{ stoat_version }}"
     force: true
-  become: true
-  become_user: "{{ stoat_service_user }}"
-  become_method: su
-  environment:
-    HOME: "/home/{{ stoat_service_user }}"
 
 - name: Ensure generate_config.sh is executable
   ansible.builtin.file:
     path: "{{ stoat_install_dir }}/generate_config.sh"
     mode: "0755"
-  become: true
+
-  become_user: "{{ stoat_service_user }}"
+# Upstream compose.yml still references docker.io/minio/minio and
-  become_method: su
+# docker.io/minio/mc, which no longer exist on Docker Hub (the pulls are
+# denied, so `podman-compose up` hangs and api/events/etc. never start).
+# MinIO's official images live on quay.io - rewrite the refs.
+- name: Point MinIO images at quay.io (docker.io/minio/* is gone)
+  ansible.builtin.replace:
+    path: "{{ stoat_install_dir }}/compose.yml"
+    regexp: 'docker\.io/minio/(minio|mc)'
+    replace: 'quay.io/minio/\1'
+
+# autumn depends on createbuckets, a one-shot that exits 0 after creating
+# the MinIO buckets. podman-compose treats `condition: service_started` as
+# "container is running", so it waits forever on the exited one-shot and
+# `podman-compose up` never completes (autumn/pushd stay Created). Use the
+# service_completed_successfully condition instead.
+- name: Fix autumn dependency on the createbuckets one-shot
+  ansible.builtin.replace:
+    path: "{{ stoat_install_dir }}/compose.yml"
+    regexp: '(createbuckets:\n\s+condition: )service_started'
+    replace: '\1service_completed_successfully'
 
 # The compose file bind-mounts these directories; create them up
-# front so rootless podman does not fail on missing sources.
+# front so podman-compose does not fail on missing sources.
 - name: Create data directories referenced by compose bind mounts
   ansible.builtin.file:
     path: "{{ stoat_install_dir }}/data/{{ item }}"
     state: directory
-    owner: "{{ stoat_service_user }}"
-    group: "{{ stoat_service_user }}"
     mode: "0755"
   loop:
     - db
@@ -50,11 +71,27 @@
   args:
     chdir: "{{ stoat_install_dir }}"
     creates: "{{ stoat_install_dir }}/Revolt.toml"
-  become: true
+
-  become_user: "{{ stoat_service_user }}"
+# Rootful podman containers run as container_t; on enforcing
-  become_method: su
+# SELinux they cannot read/write bind mounts labeled usr_t, so
-  environment:
+# register the deployment tree as container_file_t (persistent).
-    HOME: "/home/{{ stoat_service_user }}"
+- name: Ensure the deployment tree is root-owned
+  ansible.builtin.file:
+    path: "{{ stoat_install_dir }}"
+    state: directory
+    owner: root
+    group: root
+    recurse: true
+
+- name: Register the deployment directory as a container file context
+  community.general.sefcontext:
+    target: "{{ stoat_install_dir }}(/.*)?"
+    setype: container_file_t
+    state: present
+
+- name: Relabel the deployment tree with restorecon
+  ansible.builtin.command: "restorecon -RF {{ stoat_install_dir }}"
+  changed_when: false
 
 - name: Deploy podman-compose systemd unit
   ansible.builtin.template:
@@ -65,11 +102,10 @@
     mode: "0644"
 
 # First start pulls ~2 GB of images + initializes MongoDB, which can
-# take several minutes. TimeoutStartSec covers this inside the unit.
+# take several minutes. TimeoutStartSec in the unit covers this.
 - name: Enable and start the Stoat service (first pull may take minutes)
   ansible.builtin.systemd:
     name: podman-compose-stoat
     state: started
     enabled: true
-    daemon_reload: true
+    daemon_reload: true
-    timeout: 900

roles/stoat/templates/podman-compose-stoat.service.j2

@@ -6,15 +6,10 @@ Wants=network-online.target
 
 [Service]
 # oneshot + RemainAfterExit: the unit tracks compose state while
-# the containers themselves run detached under rootless podman.
+# the containers themselves run detached under the root podman.
 Type=oneshot
 RemainAfterExit=yes
-User={{ stoat_service_user }}
-Group={{ stoat_service_user }}
 WorkingDirectory={{ stoat_install_dir }}
-# Rootless podman needs the per-user runtime directory; %U expands
-# to the UID of `User=`. Lingering (enable-linger) keeps it alive.
-Environment=XDG_RUNTIME_DIR=/run/user/%U
 ExecStart=/usr/bin/podman-compose up -d --remove-orphans
 ExecStop=/usr/bin/podman-compose down
 # First start pulls ~2 GB of images; be generous.

roles/stoat/templates/sshd_hardening.conf.j2

@@ -1,8 +0,0 @@
-# Managed by Ansible - stoat role (sshd hardening drop-in)
-# Values come from group_vars/all/vars.yml
-
-PermitRootLogin {{ sshd_permit_root_login }}
-PasswordAuthentication {{ sshd_password_authentication }}
-{% if sshd_mfa_required %}
-AuthenticationMethods publickey,keyboard-interactive:pam
-{% endif %}

site.yml

@@ -1,7 +1,7 @@
 ---
 # ============================================================
 # Stoat.chat Fedora 44 server
-# Harden the host, install rootless Podman, deploy Stoat.
+# Harden the host, install Podman, deploy Stoat.
 #
 # Usage:
 #   ansible-playbook -i inventory.ini site.yml