# Power-Loss Resilience

The Klubhaus DietPi should come back fully operational after a power blip with
no human at the venue. This documents the failure mode and the hardening.

## Failure mode (2026-08-30)

Both Kuma push monitors for the Pi (`KlubHaus (ZB)`, `doorbell-listener`) stopped
reporting at 2026-08-30 21:39 UTC. The Pi's ZeroTier IP (172.30.158.123) answers
ICMP at all packet sizes (so the home→Pi ZT path is NOT an MTU black-hole), but
its LAN IP 192.168.81.147 does not answer, SSH is refused, and only
192.168.81.1 (router) + 192.168.81.134 (meshtastic) respond on the Klubhaus LAN.
Likely cause: the Pi came up but services / network / pushers did not fully
recover after a power event.

## Hardening (see `scripts/harden-boot.sh`)

| Concern | Fix |
|---|---|
| Boot hangs on interactive fsck after unclean shutdown | `fsck.repair=yes` in `/boot/cmdline.txt` |
| Kernel wedges with no recovery | `panic=10` in `/boot/cmdline.txt` |
| Hung box stays hung | RPi hardware watchdog (`dtparam=watchdog=on` + `RuntimeWatchdogSec=20`) |
| Docker / ZeroTier not started | `systemctl enable docker zerotier-one` |
| Containers don't come back | `restart: always` in compose; `klubhaus-stack.service` runs `docker compose up -d` for every compose project under `/root` after network+docker are up |
| Kuma not told the Pi is back | `kuma-online.sh` POSTs `status=up` to the push URLs in `/root/.kuma-push.env` |

## Apply

```bash
# from a machine that can SSH to the Pi (root@192.168.81.147)
just harden
# populate the Kuma push URLs once, from the Pi's existing pusher config:
#   ssh root@192.168.81.147 'vim /root/.kuma-push.env'
# then reboot to apply cmdline/config.txt changes:
just restart            # or ssh root@192.168.81.147 'reboot'
# verify the oneshot ran:
ssh root@192.168.81.147 'journalctl -u klubhaus-stack -b'
```

## Notes

- `just sync` now backs up the live Pi files before overwriting — the live
  copies are the source of truth (see AGENTS/clobber lesson).
- `scripts/kuma-online.sh` redacts the push token when it logs, so it is safe
  to run verbosely.
- The stale `192.168.9.147` references were replaced with `192.168.81.147`
  (the current Pi LAN address).