feat: power-loss boot resilience for Klubhaus DietPi

- restart: always for stack services
- harden-boot.sh: fsck.repair, panic reboot, hardware watchdog,
  docker+zerotier enabled, klubhaus-stack.service autostart
- kuma-online.sh: push up-heartbeat to Kuma on boot
- RESILIENCE.md documents the Aug 30 failure + apply steps
This commit is contained in:
2026-09-04 14:01:44 -07:00
parent ada4a22fc8
commit 2cc6193a82
4 changed files with 140 additions and 3 deletions
+47
View File
@@ -0,0 +1,47 @@
# Power-Loss Resilience
The Klubhaus DietPi should come back fully operational after a power blip with
no human at the venue. This documents the failure mode and the hardening.
## Failure mode (2026-08-30)
Both Kuma push monitors for the Pi (`KlubHaus (ZB)`, `doorbell-listener`) stopped
reporting at 2026-08-30 21:39 UTC. The Pi's ZeroTier IP (172.30.158.123) answers
ICMP at all packet sizes (so the home→Pi ZT path is NOT an MTU black-hole), but
its LAN IP 192.168.81.147 does not answer, SSH is refused, and only
192.168.81.1 (router) + 192.168.81.134 (meshtastic) respond on the Klubhaus LAN.
Likely cause: the Pi came up but services / network / pushers did not fully
recover after a power event.
## Hardening (see `scripts/harden-boot.sh`)
| Concern | Fix |
|---|---|
| Boot hangs on interactive fsck after unclean shutdown | `fsck.repair=yes` in `/boot/cmdline.txt` |
| Kernel wedges with no recovery | `panic=10` in `/boot/cmdline.txt` |
| Hung box stays hung | RPi hardware watchdog (`dtparam=watchdog=on` + `RuntimeWatchdogSec=20`) |
| Docker / ZeroTier not started | `systemctl enable docker zerotier-one` |
| Containers don't come back | `restart: always` in compose; `klubhaus-stack.service` runs `docker compose up -d` for every compose project under `/root` after network+docker are up |
| Kuma not told the Pi is back | `kuma-online.sh` POSTs `status=up` to the push URLs in `/root/.kuma-push.env` |
## Apply
```bash
# from a machine that can SSH to the Pi (root@192.168.81.147)
just harden
# populate the Kuma push URLs once, from the Pi's existing pusher config:
# ssh root@192.168.81.147 'vim /root/.kuma-push.env'
# then reboot to apply cmdline/config.txt changes:
just restart # or ssh root@192.168.81.147 'reboot'
# verify the oneshot ran:
ssh root@192.168.81.147 'journalctl -u klubhaus-stack -b'
```
## Notes
- `just sync` now backs up the live Pi files before overwriting — the live
copies are the source of truth (see AGENTS/clobber lesson).
- `scripts/kuma-online.sh` redacts the push token when it logs, so it is safe
to run verbosely.
- The stale `192.168.9.147` references were replaced with `192.168.81.147`
(the current Pi LAN address).