david
2cc6193a82
- restart: always for stack services - harden-boot.sh: fsck.repair, panic reboot, hardware watchdog, docker+zerotier enabled, klubhaus-stack.service autostart - kuma-online.sh: push up-heartbeat to Kuma on boot - RESILIENCE.md documents the Aug 30 failure + apply steps
47 lines
2.1 KiB
Markdown
47 lines
2.1 KiB
Markdown
# Power-Loss Resilience
|
|
|
|
The Klubhaus DietPi should come back fully operational after a power blip with
|
|
no human at the venue. This documents the failure mode and the hardening.
|
|
|
|
## Failure mode (2026-08-30)
|
|
|
|
Both Kuma push monitors for the Pi (`KlubHaus (ZB)`, `doorbell-listener`) stopped
|
|
reporting at 2026-08-30 21:39 UTC. The Pi's ZeroTier IP (172.30.158.123) answers
|
|
ICMP at all packet sizes (so the home→Pi ZT path is NOT an MTU black-hole), but
|
|
its LAN IP 192.168.81.147 does not answer, SSH is refused, and only
|
|
192.168.81.1 (router) + 192.168.81.134 (meshtastic) respond on the Klubhaus LAN.
|
|
Likely cause: the Pi came up but services / network / pushers did not fully
|
|
recover after a power event.
|
|
|
|
## Hardening (see `scripts/harden-boot.sh`)
|
|
|
|
| Concern | Fix |
|
|
|---|---|
|
|
| Boot hangs on interactive fsck after unclean shutdown | `fsck.repair=yes` in `/boot/cmdline.txt` |
|
|
| Kernel wedges with no recovery | `panic=10` in `/boot/cmdline.txt` |
|
|
| Hung box stays hung | RPi hardware watchdog (`dtparam=watchdog=on` + `RuntimeWatchdogSec=20`) |
|
|
| Docker / ZeroTier not started | `systemctl enable docker zerotier-one` |
|
|
| Containers don't come back | `restart: always` in compose; `klubhaus-stack.service` runs `docker compose up -d` for every compose project under `/root` after network+docker are up |
|
|
| Kuma not told the Pi is back | `kuma-online.sh` POSTs `status=up` to the push URLs in `/root/.kuma-push.env` |
|
|
|
|
## Apply
|
|
|
|
```bash
|
|
# from a machine that can SSH to the Pi (root@192.168.81.147)
|
|
just harden
|
|
# populate the Kuma push URLs once, from the Pi's existing pusher config:
|
|
# ssh root@192.168.81.147 'vim /root/.kuma-push.env'
|
|
# then reboot to apply cmdline/config.txt changes:
|
|
just restart # or ssh root@192.168.81.147 'reboot'
|
|
# verify the oneshot ran:
|
|
ssh root@192.168.81.147 'journalctl -u klubhaus-stack -b'
|
|
```
|
|
|
|
## Notes
|
|
|
|
- `just sync` now backs up the live Pi files before overwriting — the live
|
|
copies are the source of truth (see AGENTS/clobber lesson).
|
|
- `scripts/kuma-online.sh` redacts the push token when it logs, so it is safe
|
|
to run verbosely.
|
|
- The stale `192.168.9.147` references were replaced with `192.168.81.147`
|
|
(the current Pi LAN address). |