sideline/docs/Mainline Renderer + ntfy Message Queue for ESP32.md

# Mainline Renderer \+ ntfy Message Queue for ESP32
## Problem
mainline\.py does heavy work unsuitable for ESP32: 25\+ HTTPS/TLS RSS feeds, OTF font rasterization via Pillow, Google Translate API calls, and complex text layout\. Simultaneously, messages arriving on `ntfy.sh/klubhaus_terminal_mainline` need to interrupt the news ticker on the same device\.
## Architecture: Server \+ Thin Client
Split the system into two halves that are designed together\.
**Server (mainline\.py `--serve` mode, runs on any always\-on machine)**
* Reuses existing feed fetching, caching, content filtering, translation, and Pillow font rendering pipeline — no duplication\.
* Pre\-renders each headline into a 1\-bit bitmap strip (the OTF→half\-block pipeline already produces this as an intermediate step in `_render_line()`)\.
* Exposes a lightweight HTTP API the ESP32 polls\.
**ESP32 thin client (Arduino sketch)**
* Polls the mainline server for pre\-rendered headline bitmaps over plain HTTP (no TLS needed if on the same LAN)\.
* Polls `ntfy.sh/klubhaus_terminal_mainline` directly for messages, reusing the proven `NetManager::httpGet()` \+ JSON parsing pattern from DoorbellLogic (`DoorbellLogic.cpp:155-192`)\.
* Manages scrolling, gradient coloring, and glitch effects locally (cheap per\-frame GPU work)\.
* When an ntfy message arrives, the scroll is paused and the message takes over the display — same interrupt pattern as the doorbell's ALERT→DASHBOARD flow\.
## Server API (mainline repo)
New file: `serve.py` (or `--serve` mode in mainline\.py)\.
Endpoints:
* `GET /api/headlines` — returns JSON array of headline metadata: `[{"id": 0, "src": "Nature", "ts": "14:30", "width": 280, "height": 16, "bitmap": "<base64 1-bit packed>"}]`\. Bitmaps are 1\-bit\-per\-pixel, row\-major, packed 8px/byte\. The ESP32 applies gradient color locally\.
* `GET /api/config` — returns `{"count": 120, "version": "...", "mode": "news"}` so the ESP32 knows what it's getting\.
* `GET /api/health` — `{"ok": true, "last_fetch": "...", "headline_count": 120}`
The server renders at a configurable target width (e\.g\. 800px for Board 3, 320px for Boards 1/2) via a `--width` flag or query parameter\. Height is fixed per headline by the font size\.
The server refreshes feeds on a timer (reusing `_SCROLL_DUR` cadence or a longer interval), re\-renders, and serves the latest set\. The ESP32 polls `/api/headlines` periodically (e\.g\. every 60s) and swaps in the new set\.
## Render pipeline (server side)
The existing `_render_line()` in mainline\.py already does:
1. `ImageFont.truetype()` → `ImageDraw.text()` → grayscale `Image`
2. Resize to target height
3. Threshold to 1\-bit (the `thr = 80` step)
For the server, we stop at step 3 and pack the 1\-bit data into bytes instead of converting to half\-block Unicode\. This is the exact same pipeline, just with a different output format\. The `_big_wrap()` and `_lr_gradient()` logic stays on the server for layout; gradient *coloring* moves to the ESP32 (it's just an index lookup per pixel column)\.
## ESP32 client
### State machine
```warp-runnable-command
BOOT → SCROLL ⇄ MESSAGE
              ↘ OFF (inactivity timeout)
```
* **BOOT** — WiFi connect, initial headline fetch from server\.
* **SCROLL** — Vertical scroll through pre\-rendered headlines with local gradient \+ glitch\. Polls server for new headlines periodically\. Polls ntfy every 15s\.
* **MESSAGE** — ntfy message arrived\. Scroll paused, message displayed\. Auto\-dismiss after timeout or touch\-dismiss\. Returns to SCROLL\.
* **OFF** — Backlight off after inactivity (polling continues in background)\.
### ntfy integration
The ESP32 polls `https://ntfy.sh/klubhaus_terminal_mainline/json?since=20s&poll=1` on the same 15s interval as the doorbell polls its topics\. When a message event arrives:
1. Parse JSON: `{"event": "message", "title": "...", "message": "..."}`
2. Save current scroll position\.
3. Transition to MESSAGE state\.
4. Render message text using the display library's built\-in fonts (messages are short, no custom font needed)\.
5. After `MESSAGE_TIMEOUT_MS` (e\.g\. 30s) or touch, restore scroll position and resume\.
This is architecturally identical to `DoorbellLogic::onAlert()` → `dismissAlert()`, just with different content\. The ntfy polling runs independently of the server connection, so messages work even if the mainline server is offline (the device just shows the last cached headlines)\.
### Headline storage
* Board 3 (8 MB PSRAM): store all ~120 headline bitmaps in PSRAM\. At 800px × 16px × 1 bit = 1\.6 KB each → ~192 KB total\. Trivial\.
* Boards 1/2 (PSRAM TBD): at 320px × 16px = 640 bytes each → ~77 KB for 120 headlines\. Fits if PSRAM is present\. Without PSRAM, keep ~20 headlines in a ring buffer (~13 KB)\.
### Gradient coloring (local)
The 12\-step ANSI gradient in mainline\.py maps to 12 RGB565 values:
```warp-runnable-command
const uint16_t GRADIENT[] = {
    0xFFFF, // white
    0xC7FF, // pale cyan
    0x07F9, // bright cyan
    0x07C0, // bright lime
    ...
    0x0120, // deep green
    0x18C3, // near black
};
```
For each pixel column `x` in the bitmap, the ESP32 picks `GRADIENT[x * 12 / width]` if the bit is set, else background color\. This is a per\-pixel multiply \+ table lookup — fast\.
## File layout
### mainline repo
```warp-runnable-command
mainline.py          (existing, unchanged)
serve.py             (new — HTTP server, imports mainline rendering functions)
klubhaus-doorbell-hardware.md  (existing)
```
`serve.py` imports the rendering functions from mainline\.py (after refactoring them into importable form — they're currently top\-level but not wrapped in `if __name__`)\.
### klubhaus\-doorbell repo (or mainline repo under firmware/)
```warp-runnable-command
boards/esp32-mainline/
├── esp32-mainline.ino    Main sketch
├── board_config.h        Display/pin config (copy from target board)
├── secrets.h             WiFi creds + server URL
├── MainlineLogic.h/.cpp  State machine (replaces DoorbellLogic)
├── HeadlineStore.h/.cpp  Bitmap ring buffer in PSRAM
└── NtfyPoller.h/.cpp     ntfy.sh polling (extracted from DoorbellLogic pattern)
```
The display driver is reused from the target board (e\.g\. `DisplayDriverGFX` for Board 3)\. `MainlineLogic` replaces `DoorbellLogic` as the state machine but follows the same patterns\.
## Branch strategy recommendation
The work spans two repos and has clear dependency ordering\.
### Phase 1 — Finish current branch (mainline repo)
**Branch:** `feat/arduino` (current)
**Content:** Hardware spec doc\. Already done\.
**Action:** Merge to main when ready\.
### Phase 2 — Server renderer (mainline repo)
**Branch:** `feat/renderer` (branch from main after Phase 1 merges)
**Content:**
* Refactor mainline\.py rendering functions to be importable (extract from `__main__` guard)
* `serve.py` — HTTP server with `/api/headlines`, `/api/config`, `/api/health`
* Bitmap packing utility (1\-bit row\-major)
**Why a separate branch:** This changes mainline\.py's structure (refactoring for imports) and adds a new entry point\. It's a self\-contained, testable unit — you can verify the API with `curl` before touching any Arduino code\.
### Phase 3 — ESP32 client (klubhaus\-doorbell repo, or mainline repo)
**Branch:** `feat/mainline-client` in whichever repo hosts it
**Content:**
* `MainlineLogic` state machine
* `HeadlineStore` bitmap buffer
* `NtfyPoller` for `klubhaus_terminal_mainline`
* Board\-specific sketch for the target board
**Depends on:** Phase 2 (needs a running server to test against)
**Repo decision:** If you have push access to klubhaus\-doorbell, it fits naturally as a new board target alongside the existing doorbell sketches — it reuses `NetManager`, `IDisplayDriver`, and the vendored display libraries\. If not, put it under `mainline/firmware/` and vendor the shared KlubhausCore library\.
### Merge order
1. `feat/arduino` → main (hardware spec)
2. `feat/renderer` → main (server)
3. `feat/mainline-client` → main in whichever repo (ESP32 client)
Each phase is independently testable and doesn't block the other until Phase 3 needs a running server\.