klubhaus/klubhaus-doorbell
3
0
Fork 0
Code Issues 9 Pull Requests Packages Projects Releases Wiki Activity

[EPIC] Migrate doorbell delivery ntfy→MQTT (push, not poll) #9

New Issue
Open
opened 2026-08-28 23:32:18 +00:00 by david · 4 comments
david commented 2026-08-28 23:32:18 +00:00
Owner
Copy Link
Copy Source

Problem

ESP32 doorbell displays poll ntfy.sh every 15s (POLL_INTERVAL_MS). ntfy is slow/flaky (shared cloud infra, rate limits, ISP path), so ring→display latency is up to 15s+ and depends on internet health. Use case is an internet doorbell: callers ring from outside all of our networks.

Decision

Keep ntfy.sh as the internet ingress (callers/covers unchanged — it stays reachable from anywhere). Add an ntfy→MQTT bridge and convert ESPs from poll to subscribe (push). This is the LAN/instant layer, independent of ntfy/ISP health for the display path.

Topology

caller ──> ntfy.sh topics (ALERT/SILENCE/ADMIN/METRICS/STATUS_klubhaus_topic)
                │  bridge on dhg.lol (subs/polls ntfy)
                ▼
   klubhaus mosquitto (RPi rpi-zigbee-stack) ──> klubhaus/doorbell/*
                │
                ▼
        ESP32 boards (subscribe, sub-second)
  • ntfy.sh remains primary source for the remote ring path + phone push.
  • MQTT on the klubhaus Pi becomes the delivery/display path.

Tasks

Phase 1 — Recon (no system changes)

  • Discover klubhaus mosquitto endpoint (host/IP, port), auth, TLS. Pi runs rpi-zigbee-stack (Mosquitto + Zigbee2MQTT + Node-RED).
  • Confirm reachability from dhg.lol and from ESP boards over the klubhaus↔home↔AWS peer.
  • Record endpoint + credential location in this issue (never the secret).

Phase 2 — Bridge (ntfy→MQTT) on dhg.lol

  • Tiny container: subscribe/poll the five *_klubhaus_topic ntfy topics, republish to klubhaus/doorbell/{alert,silence,admin,metrics,status}.
  • Preserve title/body fields the firmware already parses (DoorbellLogic, event/message/title).
  • Deploy on dhg.lol; verify with mosquitto_sub -h <klubhaus-pi> -t 'klubhaus/doorbell/#'.

Phase 3 — Firmware (klubhaus-doorbell)

  • Add MQTT subscribe path to DoorbellLogic alongside ntfy poll; reuse onAlert/onSilence/onAdmin.
  • New config: MQTT_HOST, creds, topic root in Config.h + secrets.h.
  • Reconnect + LWT handling; keep 15s ntfy poll as fallback (config-gated).

Phase 4 — Hardening & ramp-down

  • Lock down klubhaus mosquitto (allow_anonymous false, enable pwfile) before doorbell traffic flows.
  • Once stable, gate the ntfy poll behind a config flag; ntfy stays for phone push only.

Open questions

  • Klubhaus broker host/auth (Phase 1 unblocks everything).
  • Should the frontend "ring" also publish MQTT directly? Recommended: no — keep ntfy-only for reachability from anywhere.

Verification criteria

  • Ring from a device not on any of our networks → ESP display < 1s.
  • Doorbell still rings if ntfy.sh is down (bridge caches last-known state).
  • Mosquitto rejects anonymous clients (auth enforced).

Related

  • Knox thread: doorbell system (motive)
  • ntfy.sh is retained as internet ingress + phone push — not removed.
## Problem ESP32 doorbell displays poll `ntfy.sh` every 15s (`POLL_INTERVAL_MS`). ntfy is slow/flaky (shared cloud infra, rate limits, ISP path), so ring→display latency is up to 15s+ and depends on internet health. Use case is an **internet doorbell**: callers ring from outside all of our networks. ## Decision Keep ntfy.sh as the **internet ingress** (callers/covers unchanged — it stays reachable from anywhere). Add an `ntfy→MQTT` bridge and convert ESPs from poll to **subscribe** (push). This is the LAN/instant layer, independent of ntfy/ISP health for the display path. ## Topology ``` caller ──> ntfy.sh topics (ALERT/SILENCE/ADMIN/METRICS/STATUS_klubhaus_topic) │ bridge on dhg.lol (subs/polls ntfy) ▼ klubhaus mosquitto (RPi rpi-zigbee-stack) ──> klubhaus/doorbell/* │ ▼ ESP32 boards (subscribe, sub-second) ``` - ntfy.sh remains primary source for the remote ring path + phone push. - MQTT on the klubhaus Pi becomes the delivery/display path. ## Tasks ### Phase 1 — Recon (no system changes) - [ ] Discover klubhaus mosquitto endpoint (host/IP, port), auth, TLS. Pi runs `rpi-zigbee-stack` (Mosquitto + Zigbee2MQTT + Node-RED). - [ ] Confirm reachability from dhg.lol and from ESP boards over the klubhaus↔home↔AWS peer. - [ ] Record endpoint + credential *location* in this issue (never the secret). ### Phase 2 — Bridge (`ntfy→MQTT`) on dhg.lol - [ ] Tiny container: subscribe/poll the five `*_klubhaus_topic` ntfy topics, republish to `klubhaus/doorbell/{alert,silence,admin,metrics,status}`. - [ ] Preserve title/body fields the firmware already parses (`DoorbellLogic`, event/message/title). - [ ] Deploy on dhg.lol; verify with `mosquitto_sub -h <klubhaus-pi> -t 'klubhaus/doorbell/#'`. ### Phase 3 — Firmware (`klubhaus-doorbell`) - [ ] Add MQTT subscribe path to `DoorbellLogic` alongside ntfy poll; reuse `onAlert/onSilence/onAdmin`. - [ ] New config: `MQTT_HOST`, creds, topic root in `Config.h` + `secrets.h`. - [ ] Reconnect + LWT handling; keep 15s ntfy poll as fallback (config-gated). ### Phase 4 — Hardening & ramp-down - [ ] Lock down klubhaus mosquitto (`allow_anonymous false`, enable `pwfile`) before doorbell traffic flows. - [ ] Once stable, gate the ntfy poll behind a config flag; ntfy stays for phone push only. ## Open questions - Klubhaus broker host/auth (Phase 1 unblocks everything). - Should the frontend "ring" also publish MQTT directly? Recommended: no — keep ntfy-only for reachability from anywhere. ## Verification criteria - Ring from a device **not** on any of our networks → ESP display < 1s. - Doorbell still rings if ntfy.sh is down (bridge caches last-known state). - Mosquitto rejects anonymous clients (auth enforced). ## Related - Knox thread: doorbell system (motive) - ntfy.sh is retained as internet ingress + phone push — not removed.
david commented 2026-08-28 23:33:18 +00:00
Author
Owner
Copy Link
Copy Source

Phase 1 recon results

Mosquitto config (from david/rpi-zigbee-stack/mosquitto.conf, cloned 2026-08-28):

  • listener 1883, allow_anonymous true, no TLS, no pwfile → bare, LAN-only profile
  • RPi runs DietPi 64-bit; stack = mosquitto + zigbee2mqtt (BLZ fork) + nodered
  • Introspection blocked for now (see below)

Network topology (confirmed):

  • Home (this host): 192.168.8.111 (wlp2s0), gateway 192.168.8.1 = berylpunk
  • Klubhaus reached via ZeroTier ztbpapydnj (local 172.30.53.219/16)
  • Static route: 192.168.9.0/24 via 172.30.68.103 dev ztbpapydnj
  • ZeroTier peer alive: ~2.4ms

Pi address — README is stale: README says 192.168.9.147; known_hosts has SSH keys for 192.168.9.167. Both currently unresponsive:

  • .147: ICMP alive (only host responding in scan was .1), MQTT :1883 refused
  • .167: ICMP no answer, SSH :22 refused
  • Full ICMP sweep of 192.168.9.0/24 → only .1 alive

Blocker: klubhaus Pi is not reachable from this host right now. Need Zebra/tunnel access on-site, or the Pi powered on/reconnected before we can introspect the broker and test the bridge.

Action on unblock: SSH to Pi, verify mosquitto up (systemctl/docker compose), capture real host/IP + pwfile presence, then Phase 2 (bridge on dhg.lol) can proceed independently of the Pi being reachable for build/test.

## Phase 1 recon results **Mosquitto config (from `david/rpi-zigbee-stack/mosquitto.conf`, cloned 2026-08-28):** - `listener 1883`, `allow_anonymous true`, no TLS, no pwfile → **bare, LAN-only profile** - RPi runs DietPi 64-bit; stack = mosquitto + zigbee2mqtt (BLZ fork) + nodered - Introspection blocked for now (see below) **Network topology (confirmed):** - Home (this host): `192.168.8.111` (wlp2s0), gateway `192.168.8.1` = berylpunk - Klubhaus reached via **ZeroTier** `ztbpapydnj` (local `172.30.53.219/16`) - Static route: `192.168.9.0/24 via 172.30.68.103 dev ztbpapydnj` - ZeroTier peer alive: ~2.4ms **Pi address — README is stale:** README says `192.168.9.147`; `known_hosts` has SSH keys for `192.168.9.167`. Both currently **unresponsive**: - `.147`: ICMP alive (only host responding in scan was `.1`), MQTT `:1883` refused - `.167`: ICMP no answer, SSH `:22` refused - Full ICMP sweep of `192.168.9.0/24` → only `.1` alive **Blocker:** klubhaus Pi is not reachable from this host right now. Need Zebra/tunnel access on-site, or the Pi powered on/reconnected before we can introspect the broker and test the bridge. **Action on unblock:** SSH to Pi, verify mosquitto up (`systemctl`/docker compose), capture real host/IP + pwfile presence, then Phase 2 (bridge on dhg.lol) can proceed independently of the Pi being reachable for build/test.
david commented 2026-08-28 23:45:46 +00:00
Author
Owner
Copy Link
Copy Source

Plan correction — the bridge already exists (DietPi side)

Phase 1 recon completed by reaching the Pi at its real address root@192.168.81.147 (ZeroTier peer 172.30.158.123; earlier .9.147/.167 addresses are stale). Unblocked by re-running the recorded zt-mtu-probe.sh (the zerotier-mtu.service was enabled-but-dead).

Discovered: doorbell-listener on the Pi already does ntfy→MQTT

Source: /root/nr-flow-validator/infra/doorbell-listener/doorbell-listener.py (image infra-doorbell-listener).

  • Polls ALERT_klubhaus_topic[_test] every 30s (paho v2 API), on fresh alert publishes {"state":"TOGGLE"} to zigbee2mqtt/Sideboard Lamp/set (flash the lamp). Canary/probe alerts tagged doorbell-probe → zigbee2mqtt/DoorbellProbe/set instead.
  • BUG: getting 429 Too Many Requests from ntfy.sh — polls without the auth token (anonymous tier), 2 topics × 30s ≈ 5,760 req/day ≫ anonymous limit. This is the primary cause of observed doorbell slowness AND of the Kuma doorbell-listener (monitor #25) "No heartbeat" history.
  • This is the ntfy→MQTT element I planned to build on dhg.lol — not needed; the listener already exists. New plan below.

Revised Phase 2 (replaces old build-on-dhg.lol)

  • Add NTFY_TOKEN env + auth= param to doorbell-listener.py poll URL (mirror frontend's Bearer tk_... handling). Fixes 429.
  • Extend listener to republish alerts to klubhaus/doorbell/alert (plus silence/admin) so ESPs can subscribe.
  • Verify with mosquitto_sub -h 192.168.81.147 -t 'klubhaus/doorbell/#'.

Revised Phase 3 (firmware)

  • Add MQTT subscribe path (PubSubClient) to DoorbellLogic, keep ntfy poll fallback.

Mosquitto (192.168.81.147:1883): allow_anonymous true, has password_file + acl.conf present but anonymous still open. Route to clubhouse restored via MTU probe.

Pending: confirm ntfy token for the listener and whether NOTE: the frontend's embedded token (tk_sw5...) should be reused server-side.

## Plan correction — the bridge already exists (DietPi side) Phase 1 recon completed by reaching the Pi at its **real** address `root@192.168.81.147` (ZeroTier peer `172.30.158.123`; earlier `.9.147`/`.167` addresses are stale). Unblocked by re-running the recorded `zt-mtu-probe.sh` (the `zerotier-mtu.service` was enabled-but-dead). ### Discovered: `doorbell-listener` on the Pi already does ntfy→MQTT Source: `/root/nr-flow-validator/infra/doorbell-listener/doorbell-listener.py` (image `infra-doorbell-listener`). - Polls `ALERT_klubhaus_topic[_test]` every 30s (paho v2 API), on fresh alert publishes `{"state":"TOGGLE"}` to `zigbee2mqtt/Sideboard Lamp/set` (flash the lamp). Canary/probe alerts tagged `doorbell-probe` → `zigbee2mqtt/DoorbellProbe/set` instead. - **BUG: getting `429 Too Many Requests` from ntfy.sh** — polls without the `auth` token (anonymous tier), 2 topics × 30s ≈ 5,760 req/day ≫ anonymous limit. This is the primary cause of observed doorbell slowness AND of the Kuma `doorbell-listener` (monitor #25) "No heartbeat" history. - This is the ntfy→MQTT element I planned to build on dhg.lol — not needed; the listener already exists. New plan below. ### Revised Phase 2 (replaces old build-on-dhg.lol) - [ ] Add `NTFY_TOKEN` env + `auth=` param to `doorbell-listener.py` poll URL (mirror frontend's `Bearer tk_...` handling). Fixes 429. - [ ] Extend listener to republish alerts to `klubhaus/doorbell/alert` (plus silence/admin) so ESPs can subscribe. - [ ] Verify with `mosquitto_sub -h 192.168.81.147 -t 'klubhaus/doorbell/#'`. ### Revised Phase 3 (firmware) - [ ] Add MQTT subscribe path (PubSubClient) to `DoorbellLogic`, keep ntfy poll fallback. Mosquitto (`192.168.81.147:1883`): `allow_anonymous true`, has `password_file` + `acl.conf` present but anonymous still open. Route to clubhouse restored via MTU probe. Pending: confirm ntfy token for the listener and whether NOTE: the frontend's embedded token (`tk_sw5...`) should be reused server-side.
david commented 2026-08-28 23:53:48 +00:00
Author
Owner
Copy Link
Copy Source

429 fix shipped + probe routing regression fixed

Root cause of slowness confirmed: doorbell-listener polled ntfy.sh without the auth token → anonymous-tier 429s (≈2 topics × every 5s). Fixed in nr-flow-validator/infra/doorbell-listener/doorbell-listener.py:

  • Added NTFY_TOKEN env + auth_param() (base64 Bearer tk_..., mirrors frontend buildAuthParam)
  • Poll URL now: json?poll=1&auth=<b64>&since=<last_id>
  • NTFY_TOKEN set in infra/compose.yaml (same token as frontend — topics are access-controlled)

Regression caught during deploy: the tracked nr-flow-validator copy of doorbell-listener.py was BEHIND the Pi's live file — the Pi had uncommitted probe-routing support (PROBE_PLUG_TOPIC → zigbee2mqtt/DoorbellProbe/set for doorbell-probe-tagged alerts so E2E tests never flash the real lamp). Deploying the tracked copy clobbered it once (one real Squiggle lamp flash). Fixed by merging the Pi version + token change; tracked repo copy now equals the Pi's live file.

Verified:

  • No 429s since rebuild (auth on every poll)
  • Test ring → ALERT_klubhaus_topic_test received in 3s → TOGGLE flash to zigbee2mqtt/DoorbellProbe/set (probe device only)
  • Container infra-doorbell-listener rebuilt with merged code (2 probe refs, 3 auth refs)

Note for future sessions: doorbell-listener.py has uncommitted local state on the Pi in nr-flow-validator (M doorbell-listener.py, M compose.yaml, ?? doorbell-probe/, ?? scripts/). Sync repo ↔ Pi before editing; the Pi copy may be ahead.

## 429 fix shipped + probe routing regression fixed **Root cause of slowness confirmed:** `doorbell-listener` polled ntfy.sh *without* the `auth` token → anonymous-tier 429s (≈2 topics × every 5s). Fixed in `nr-flow-validator/infra/doorbell-listener/doorbell-listener.py`: - Added `NTFY_TOKEN` env + `auth_param()` (base64 `Bearer tk_...`, mirrors frontend `buildAuthParam`) - Poll URL now: `json?poll=1&auth=<b64>&since=<last_id>` - `NTFY_TOKEN` set in `infra/compose.yaml` (same token as frontend — topics are access-controlled) **Regression caught during deploy:** the tracked `nr-flow-validator` copy of `doorbell-listener.py` was BEHIND the Pi's live file — the Pi had uncommitted probe-routing support (`PROBE_PLUG_TOPIC` → zigbee2mqtt/DoorbellProbe/set for `doorbell-probe`-tagged alerts so E2E tests never flash the real lamp). Deploying the tracked copy clobbered it once (one real Squiggle lamp flash). Fixed by merging the Pi version + token change; tracked repo copy now equals the Pi's live file. **Verified:** - No 429s since rebuild (auth on every poll) - Test ring → `ALERT_klubhaus_topic_test` received in 3s → TOGGLE flash to `zigbee2mqtt/DoorbellProbe/set` (probe device only) - Container `infra-doorbell-listener` rebuilt with merged code (2 probe refs, 3 auth refs) **Note for future sessions:** `doorbell-listener.py` has uncommitted local state on the Pi in `nr-flow-validator` (`M doorbell-listener.py`, `M compose.yaml`, `?? doorbell-probe/`, `?? scripts/`). Sync repo ↔ Pi before editing; the Pi copy may be ahead.
david commented 2026-08-29 05:17:14 +00:00
Author
Owner
Copy Link
Copy Source

Kuma doorbell service down — fixed + committed

Root cause: the doorbell-kuma-heartbeat.sh canary published to ntfy.sh anonymously → anonymous rate limit → ntfy_publish_failed → pushed status=down to Kuma monitor #25. Compounding: the listener polled ntfy.sh every 5s (~34k req/day), exhausting the account window.

Fixes (live on DietPi, committed to 64906):

  • doorbell-listener.py: poll with auth= (Bearer token) + probe-tag routing → DoorbellProbe
  • kuma-heartbeat.sh: canary publishes Authorization: Bearer <token> instead of anonymous; token read from .secrets/ntfy-token
  • Composed POLL_INTERVAL raised 5s → 60s

Verified: Kuma #25 heartbeats status=1 | doorbell ok; listener logs 0× 429.

Commit: f4d6be0 doorbell: add ntfy token auth to listener poll + authenticated kuma canary (nr-flow-validator).

## Kuma doorbell service down — fixed + committed **Root cause:** the `doorbell-kuma-heartbeat.sh` canary published to ntfy.sh **anonymously** → anonymous rate limit → `ntfy_publish_failed` → pushed `status=down` to Kuma monitor #25. Compounding: the listener polled ntfy.sh every 5s (~34k req/day), exhausting the account window. **Fixes (live on DietPi, committed to `64906`):** - `doorbell-listener.py`: poll with `auth=` (Bearer token) + probe-tag routing → DoorbellProbe - `kuma-heartbeat.sh`: canary publishes `Authorization: Bearer <token>` instead of anonymous; token read from `.secrets/ntfy-token` - Composed `POLL_INTERVAL` raised 5s → 60s **Verified:** Kuma #25 heartbeats `status=1 | doorbell ok`; listener logs 0× 429. Commit: `f4d6be0 doorbell: add ntfy token auth to listener poll + authenticated kuma canary` (nr-flow-validator).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
critical
enhancement
warning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
datapotatobaby david genewildish
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: klubhaus/klubhaus-doorbell#9
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?