david
2cc6193a82
- restart: always for stack services - harden-boot.sh: fsck.repair, panic reboot, hardware watchdog, docker+zerotier enabled, klubhaus-stack.service autostart - kuma-online.sh: push up-heartbeat to Kuma on boot - RESILIENCE.md documents the Aug 30 failure + apply steps
2.1 KiB
2.1 KiB
Power-Loss Resilience
The Klubhaus DietPi should come back fully operational after a power blip with no human at the venue. This documents the failure mode and the hardening.
Failure mode (2026-08-30)
Both Kuma push monitors for the Pi (KlubHaus (ZB), doorbell-listener) stopped
reporting at 2026-08-30 21:39 UTC. The Pi's ZeroTier IP (172.30.158.123) answers
ICMP at all packet sizes (so the home→Pi ZT path is NOT an MTU black-hole), but
its LAN IP 192.168.81.147 does not answer, SSH is refused, and only
192.168.81.1 (router) + 192.168.81.134 (meshtastic) respond on the Klubhaus LAN.
Likely cause: the Pi came up but services / network / pushers did not fully
recover after a power event.
Hardening (see scripts/harden-boot.sh)
| Concern | Fix |
|---|---|
| Boot hangs on interactive fsck after unclean shutdown | fsck.repair=yes in /boot/cmdline.txt |
| Kernel wedges with no recovery | panic=10 in /boot/cmdline.txt |
| Hung box stays hung | RPi hardware watchdog (dtparam=watchdog=on + RuntimeWatchdogSec=20) |
| Docker / ZeroTier not started | systemctl enable docker zerotier-one |
| Containers don't come back | restart: always in compose; klubhaus-stack.service runs docker compose up -d for every compose project under /root after network+docker are up |
| Kuma not told the Pi is back | kuma-online.sh POSTs status=up to the push URLs in /root/.kuma-push.env |
Apply
# from a machine that can SSH to the Pi (root@192.168.81.147)
just harden
# populate the Kuma push URLs once, from the Pi's existing pusher config:
# ssh root@192.168.81.147 'vim /root/.kuma-push.env'
# then reboot to apply cmdline/config.txt changes:
just restart # or ssh root@192.168.81.147 'reboot'
# verify the oneshot ran:
ssh root@192.168.81.147 'journalctl -u klubhaus-stack -b'
Notes
just syncnow backs up the live Pi files before overwriting — the live copies are the source of truth (see AGENTS/clobber lesson).scripts/kuma-online.shredacts the push token when it logs, so it is safe to run verbosely.- The stale
192.168.9.147references were replaced with192.168.81.147(the current Pi LAN address).