feat: power-loss boot resilience for Klubhaus DietPi
- restart: always for stack services - harden-boot.sh: fsck.repair, panic reboot, hardware watchdog, docker+zerotier enabled, klubhaus-stack.service autostart - kuma-online.sh: push up-heartbeat to Kuma on boot - RESILIENCE.md documents the Aug 30 failure + apply steps
This commit is contained in:
@@ -0,0 +1,47 @@
|
||||
# Power-Loss Resilience
|
||||
|
||||
The Klubhaus DietPi should come back fully operational after a power blip with
|
||||
no human at the venue. This documents the failure mode and the hardening.
|
||||
|
||||
## Failure mode (2026-08-30)
|
||||
|
||||
Both Kuma push monitors for the Pi (`KlubHaus (ZB)`, `doorbell-listener`) stopped
|
||||
reporting at 2026-08-30 21:39 UTC. The Pi's ZeroTier IP (172.30.158.123) answers
|
||||
ICMP at all packet sizes (so the home→Pi ZT path is NOT an MTU black-hole), but
|
||||
its LAN IP 192.168.81.147 does not answer, SSH is refused, and only
|
||||
192.168.81.1 (router) + 192.168.81.134 (meshtastic) respond on the Klubhaus LAN.
|
||||
Likely cause: the Pi came up but services / network / pushers did not fully
|
||||
recover after a power event.
|
||||
|
||||
## Hardening (see `scripts/harden-boot.sh`)
|
||||
|
||||
| Concern | Fix |
|
||||
|---|---|
|
||||
| Boot hangs on interactive fsck after unclean shutdown | `fsck.repair=yes` in `/boot/cmdline.txt` |
|
||||
| Kernel wedges with no recovery | `panic=10` in `/boot/cmdline.txt` |
|
||||
| Hung box stays hung | RPi hardware watchdog (`dtparam=watchdog=on` + `RuntimeWatchdogSec=20`) |
|
||||
| Docker / ZeroTier not started | `systemctl enable docker zerotier-one` |
|
||||
| Containers don't come back | `restart: always` in compose; `klubhaus-stack.service` runs `docker compose up -d` for every compose project under `/root` after network+docker are up |
|
||||
| Kuma not told the Pi is back | `kuma-online.sh` POSTs `status=up` to the push URLs in `/root/.kuma-push.env` |
|
||||
|
||||
## Apply
|
||||
|
||||
```bash
|
||||
# from a machine that can SSH to the Pi (root@192.168.81.147)
|
||||
just harden
|
||||
# populate the Kuma push URLs once, from the Pi's existing pusher config:
|
||||
# ssh root@192.168.81.147 'vim /root/.kuma-push.env'
|
||||
# then reboot to apply cmdline/config.txt changes:
|
||||
just restart # or ssh root@192.168.81.147 'reboot'
|
||||
# verify the oneshot ran:
|
||||
ssh root@192.168.81.147 'journalctl -u klubhaus-stack -b'
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- `just sync` now backs up the live Pi files before overwriting — the live
|
||||
copies are the source of truth (see AGENTS/clobber lesson).
|
||||
- `scripts/kuma-online.sh` redacts the push token when it logs, so it is safe
|
||||
to run verbosely.
|
||||
- The stale `192.168.9.147` references were replaced with `192.168.81.147`
|
||||
(the current Pi LAN address).
|
||||
Reference in New Issue
Block a user