feat: power-loss boot resilience for Klubhaus DietPi

- restart: always for stack services
- harden-boot.sh: fsck.repair, panic reboot, hardware watchdog,
  docker+zerotier enabled, klubhaus-stack.service autostart
- kuma-online.sh: push up-heartbeat to Kuma on boot
- RESILIENCE.md documents the Aug 30 failure + apply steps

RESILIENCE.md

@@ -0,0 +1,47 @@
+# Power-Loss Resilience
+
+The Klubhaus DietPi should come back fully operational after a power blip with
+no human at the venue. This documents the failure mode and the hardening.
+
+## Failure mode (2026-08-30)
+
+Both Kuma push monitors for the Pi (`KlubHaus (ZB)`, `doorbell-listener`) stopped
+reporting at 2026-08-30 21:39 UTC. The Pi's ZeroTier IP (172.30.158.123) answers
+ICMP at all packet sizes (so the home→Pi ZT path is NOT an MTU black-hole), but
+its LAN IP 192.168.81.147 does not answer, SSH is refused, and only
+192.168.81.1 (router) + 192.168.81.134 (meshtastic) respond on the Klubhaus LAN.
+Likely cause: the Pi came up but services / network / pushers did not fully
+recover after a power event.
+
+## Hardening (see `scripts/harden-boot.sh`)
+
+| Concern | Fix |
+|---|---|
+| Boot hangs on interactive fsck after unclean shutdown | `fsck.repair=yes` in `/boot/cmdline.txt` |
+| Kernel wedges with no recovery | `panic=10` in `/boot/cmdline.txt` |
+| Hung box stays hung | RPi hardware watchdog (`dtparam=watchdog=on` + `RuntimeWatchdogSec=20`) |
+| Docker / ZeroTier not started | `systemctl enable docker zerotier-one` |
+| Containers don't come back | `restart: always` in compose; `klubhaus-stack.service` runs `docker compose up -d` for every compose project under `/root` after network+docker are up |
+| Kuma not told the Pi is back | `kuma-online.sh` POSTs `status=up` to the push URLs in `/root/.kuma-push.env` |
+
+## Apply
+
+```bash
+# from a machine that can SSH to the Pi (root@192.168.81.147)
+just harden
+# populate the Kuma push URLs once, from the Pi's existing pusher config:
+#   ssh root@192.168.81.147 'vim /root/.kuma-push.env'
+# then reboot to apply cmdline/config.txt changes:
+just restart            # or ssh root@192.168.81.147 'reboot'
+# verify the oneshot ran:
+ssh root@192.168.81.147 'journalctl -u klubhaus-stack -b'
+```
+
+## Notes
+
+- `just sync` now backs up the live Pi files before overwriting — the live
+  copies are the source of truth (see AGENTS/clobber lesson).
+- `scripts/kuma-online.sh` redacts the push token when it logs, so it is safe
+  to run verbosely.
+- The stale `192.168.9.147` references were replaced with `192.168.81.147`
+  (the current Pi LAN address).

compose.yaml

@@ -2,7 +2,7 @@ services:
   mosquitto:
     image: eclipse-mosquitto:latest
     container_name: mosquitto
-    restart: unless-stopped
+    restart: always
     user: "1883:1883"
     ports:
       - "1883:1883"
@@ -14,7 +14,7 @@ services:
   zigbee2mqtt:
     image: ghcr.io/pine64/zigbee2mqtt:latest-dev   # BLZ fork, NOT koenkk
     container_name: zigbee2mqtt
-    restart: unless-stopped
+    restart: always
     depends_on:
       - mosquitto
     ports:
@@ -41,7 +41,7 @@ services:
   nodered:
     image: nodered/node-red:latest
     container_name: nodered
-    restart: unless-stopped
+    restart: always
     depends_on:
       - mosquitto
     user: "1000:1000"

justfile

@@ -1,7 +1,7 @@
 # rpi-zigbee-stack — Docker Compose stack management for RPi 3B+
 
 set export
-RPI := "root@192.168.9.147"
+RPI := "root@192.168.81.147"
 COMPOSE := "/root/compose.yaml"
 
 # Default: show stack status
@@ -66,21 +66,35 @@ zb-devices:
 
 # Open Node-RED editor
 nr-editor:
-    xdg-open http://192.168.9.147:1880 2>/dev/null || open http://192.168.9.147:1880 2>/dev/null
+    xdg-open http://192.168.81.147:1880 2>/dev/null || open http://192.168.81.147:1880 2>/dev/null
 
 # Open Node-RED dashboard
 nr-dashboard:
-    xdg-open http://192.168.9.147:1880/ui 2>/dev/null || open http://192.168.9.147:1880/ui 2>/dev/null
+    xdg-open http://192.168.81.147:1880/ui 2>/dev/null || open http://192.168.81.147:1880/ui 2>/dev/null
 
 # ---- Maintenance ----
 
-# Sync compose.yaml and configs to RPi
+# Sync compose.yaml and configs to RPi (backs up live files first)
 sync:
+    ssh {{RPI}} "cp /root/compose.yaml /root/compose.yaml.bak.$$(date +%s) 2>/dev/null || true"
+    ssh {{RPI}} "cp /root/mosquitto/config/mosquitto.conf /root/mosquitto/config/mosquitto.conf.bak.$$(date +%s) 2>/dev/null || true"
+    ssh {{RPI}} "cp /root/zigbee2mqtt/configuration.yaml /root/zigbee2mqtt/configuration.yaml.bak.$$(date +%s) 2>/dev/null || true"
     scp compose.yaml {{RPI}}:/root/compose.yaml
     scp mosquitto.conf {{RPI}}:/root/mosquitto/config/mosquitto.conf
     scp zigbee2mqtt.yaml {{RPI}}:/root/zigbee2mqtt/configuration.yaml
     @echo "Configs synced to RPi. Run 'just restart' to apply."
 
+# Apply boot-resilience hardening (fsck auto-repair, watchdog, stack autostart)
+harden:
+    ssh {{RPI}} "cp /root/harden-boot.sh /root/harden-boot.sh.bak.$$(date +%s) 2>/dev/null || true"
+    scp scripts/harden-boot.sh {{RPI}}:/root/harden-boot.sh
+    scp scripts/kuma-online.sh {{RPI}}:/root/kuma-online.sh
+    ssh {{RPI}} "bash /root/harden-boot.sh"
+
+# Push an immediate "online" heartbeat to Kuma from the RPi
+online:
+    ssh {{RPI}} "bash /usr/local/sbin/kuma-online.sh"
+
 # Full redeploy: sync configs + restart
 redeploy: sync restart
 

scripts/harden-boot.sh

@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Idempotent boot-resilience hardening for the Klubhaus DietPi (run as root).
+# After a power blip the Pi should boot cleanly, repair its fs, restart the
+# stack, and mark itself online in Kuma without human intervention.
+
+BOOT="${BOOT:-/boot}"
+SVC=/etc/systemd/system/klubhaus-stack.service
+BRINGUP=/usr/local/sbin/klubhaus-bringup.sh
+ONLINE=/usr/local/sbin/kuma-online.sh
+
+echo "[1/6] cmdline: fsck.repair + panic reboot"
+if [ -f "$BOOT/cmdline.txt" ]; then
+    for opt in fsck.repair=yes panic=10; do
+        grep -q "$opt" "$BOOT/cmdline.txt" || sed -i "s/$/ $opt/" "$BOOT/cmdline.txt"
+    done
+    echo "  -> $(cat "$BOOT/cmdline.txt")"
+fi
+
+echo "[2/6] watchdog"
+if [ -f "$BOOT/config.txt" ]; then
+    grep -q "dtparam=watchdog=on" "$BOOT/config.txt" || echo "dtparam=watchdog=on" >> "$BOOT/config.txt"
+fi
+sed -i 's/^#RuntimeWatchdogSec=.*/RuntimeWatchdogSec=20/' /etc/systemd/system.conf
+grep -q "^RuntimeWatchdogSec=" /etc/systemd/system.conf || echo "RuntimeWatchdogSec=20" >> /etc/systemd/system.conf
+
+echo "[3/6] enable docker + zerotier"
+systemctl enable docker.service zerotier-one.service 2>/dev/null || true
+
+echo "[4/6] install bringup script"
+cat > "$BRINGUP" <<'EOF'
+#!/usr/bin/env bash
+set -euo pipefail
+while IFS= read -r f; do
+    docker compose -f "$f" up -d
+done < <(find /root -maxdepth 2 -iname 'compose.y*ml' -print 2>/dev/null)
+/usr/local/sbin/kuma-online.sh || true
+EOF
+chmod +x "$BRINGUP"
+
+echo "[5/6] install kuma-online hook"
+if [ -f /root/kuma-online.sh ]; then
+    cp /root/kuma-online.sh "$ONLINE"
+    chmod +x "$ONLINE"
+    echo "  -> installed from /root/kuma-online.sh"
+else
+    echo "  -> WARN: /root/kuma-online.sh not found; skipping (pushers already push)"
+fi
+
+echo "[6/6] install + enable stack service"
+cat > "$SVC" <<'EOF'
+[Unit]
+Description=Bring up Klubhaus docker compose stack and mark online in Kuma
+After=network-online.target docker.service
+Wants=network-online.target docker.service
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/usr/local/sbin/klubhaus-bringup.sh
+
+[Install]
+WantedBy=multi-user.target
+EOF
+systemctl daemon-reload
+systemctl enable klubhaus-stack.service
+
+echo
+echo "Done. A reboot applies cmdline/config.txt changes (fsck + watchdog)."
+echo "Verify after reboot: journalctl -u klubhaus-stack -b"

scripts/kuma-online.sh

@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Push an "up" heartbeat to every configured Kuma push URL.
+# URLs live in /root/.kuma-push.env (one per line, # for comments).
+ENV_FILE=/root/.kuma-push.env
+
+[ -f "$ENV_FILE" ] || { echo "no $ENV_FILE"; exit 0; }
+
+while IFS= read -r url; do
+    case "$url" in
+        ""|"#"*) continue ;;
+    esac
+    if curl -fsS -G --data-urlencode "status=up" --data-urlencode "msg=pi-online" "$url" >/dev/null 2>&1; then
+        echo "OK  ${url#https://up.notsosm.art/api/push/}"
+    else
+        echo "FAIL ${url#https://up.notsosm.art/api/push/}"
+    fi
+done < "$ENV_FILE"