Merge branch 'main' into feat/scalability

Reviewed-on: #14

Mainline Renderer + ntfy Message Queue for ESP32.md

@@ -3,29 +3,29 @@
 mainline\.py does heavy work unsuitable for ESP32: 25\+ HTTPS/TLS RSS feeds, OTF font rasterization via Pillow, Google Translate API calls, and complex text layout\. Simultaneously, messages arriving on `ntfy.sh/klubhaus_terminal_mainline` need to interrupt the news ticker on the same device\.
 ## Architecture: Server \+ Thin Client
 Split the system into two halves that are designed together\.
-**Server \(mainline\.py `--serve` mode, runs on any always\-on machine\)**
+**Server (mainline\.py `--serve` mode, runs on any always\-on machine)**
 * Reuses existing feed fetching, caching, content filtering, translation, and Pillow font rendering pipeline — no duplication\.
-* Pre\-renders each headline into a 1\-bit bitmap strip \(the OTF→half\-block pipeline already produces this as an intermediate step in `_render_line()`\)\.
+* Pre\-renders each headline into a 1\-bit bitmap strip (the OTF→half\-block pipeline already produces this as an intermediate step in `_render_line()`)\.
 * Exposes a lightweight HTTP API the ESP32 polls\.
-**ESP32 thin client \(Arduino sketch\)**
+**ESP32 thin client (Arduino sketch)**
-* Polls the mainline server for pre\-rendered headline bitmaps over plain HTTP \(no TLS needed if on the same LAN\)\.
+* Polls the mainline server for pre\-rendered headline bitmaps over plain HTTP (no TLS needed if on the same LAN)\.
-* Polls `ntfy.sh/klubhaus_terminal_mainline` directly for messages, reusing the proven `NetManager::httpGet()` \+ JSON parsing pattern from DoorbellLogic \(`DoorbellLogic.cpp:155-192`\)\.
+* Polls `ntfy.sh/klubhaus_terminal_mainline` directly for messages, reusing the proven `NetManager::httpGet()` \+ JSON parsing pattern from DoorbellLogic (`DoorbellLogic.cpp:155-192`)\.
-* Manages scrolling, gradient coloring, and glitch effects locally \(cheap per\-frame GPU work\)\.
+* Manages scrolling, gradient coloring, and glitch effects locally (cheap per\-frame GPU work)\.
 * When an ntfy message arrives, the scroll is paused and the message takes over the display — same interrupt pattern as the doorbell's ALERT→DASHBOARD flow\.
-## Server API \(mainline repo\)
+## Server API (mainline repo)
-New file: `serve.py` \(or `--serve` mode in mainline\.py\)\.
+New file: `serve.py` (or `--serve` mode in mainline\.py)\.
 Endpoints:
 * `GET /api/headlines` — returns JSON array of headline metadata: `[{"id": 0, "src": "Nature", "ts": "14:30", "width": 280, "height": 16, "bitmap": "<base64 1-bit packed>"}]`\. Bitmaps are 1\-bit\-per\-pixel, row\-major, packed 8px/byte\. The ESP32 applies gradient color locally\.
 * `GET /api/config` — returns `{"count": 120, "version": "...", "mode": "news"}` so the ESP32 knows what it's getting\.
 * `GET /api/health` — `{"ok": true, "last_fetch": "...", "headline_count": 120}`
-The server renders at a configurable target width \(e\.g\. 800px for Board 3, 320px for Boards 1/2\) via a `--width` flag or query parameter\. Height is fixed per headline by the font size\.
+The server renders at a configurable target width (e\.g\. 800px for Board 3, 320px for Boards 1/2) via a `--width` flag or query parameter\. Height is fixed per headline by the font size\.
-The server refreshes feeds on a timer \(reusing `_SCROLL_DUR` cadence or a longer interval\), re\-renders, and serves the latest set\. The ESP32 polls `/api/headlines` periodically \(e\.g\. every 60s\) and swaps in the new set\.
+The server refreshes feeds on a timer (reusing `_SCROLL_DUR` cadence or a longer interval), re\-renders, and serves the latest set\. The ESP32 polls `/api/headlines` periodically (e\.g\. every 60s) and swaps in the new set\.
-## Render pipeline \(server side\)
+## Render pipeline (server side)
 The existing `_render_line()` in mainline\.py already does:
 1. `ImageFont.truetype()` → `ImageDraw.text()` → grayscale `Image`
 2. Resize to target height
-3. Threshold to 1\-bit \(the `thr = 80` step\)
+3. Threshold to 1\-bit (the `thr = 80` step)
-For the server, we stop at step 3 and pack the 1\-bit data into bytes instead of converting to half\-block Unicode\. This is the exact same pipeline, just with a different output format\. The `_big_wrap()` and `_lr_gradient()` logic stays on the server for layout; gradient *coloring* moves to the ESP32 \(it's just an index lookup per pixel column\)\.
+For the server, we stop at step 3 and pack the 1\-bit data into bytes instead of converting to half\-block Unicode\. This is the exact same pipeline, just with a different output format\. The `_big_wrap()` and `_lr_gradient()` logic stays on the server for layout; gradient *coloring* moves to the ESP32 (it's just an index lookup per pixel column)\.
 ## ESP32 client
 ### State machine
 ```warp-runnable-command
@@ -35,19 +35,19 @@ BOOT → SCROLL ⇄ MESSAGE
 * **BOOT** — WiFi connect, initial headline fetch from server\.
 * **SCROLL** — Vertical scroll through pre\-rendered headlines with local gradient \+ glitch\. Polls server for new headlines periodically\. Polls ntfy every 15s\.
 * **MESSAGE** — ntfy message arrived\. Scroll paused, message displayed\. Auto\-dismiss after timeout or touch\-dismiss\. Returns to SCROLL\.
-* **OFF** — Backlight off after inactivity \(polling continues in background\)\.
+* **OFF** — Backlight off after inactivity (polling continues in background)\.
 ### ntfy integration
 The ESP32 polls `https://ntfy.sh/klubhaus_terminal_mainline/json?since=20s&poll=1` on the same 15s interval as the doorbell polls its topics\. When a message event arrives:
 1. Parse JSON: `{"event": "message", "title": "...", "message": "..."}`
 2. Save current scroll position\.
 3. Transition to MESSAGE state\.
-4. Render message text using the display library's built\-in fonts \(messages are short, no custom font needed\)\.
+4. Render message text using the display library's built\-in fonts (messages are short, no custom font needed)\.
-5. After `MESSAGE_TIMEOUT_MS` \(e\.g\. 30s\) or touch, restore scroll position and resume\.
+5. After `MESSAGE_TIMEOUT_MS` (e\.g\. 30s) or touch, restore scroll position and resume\.
-This is architecturally identical to `DoorbellLogic::onAlert()` → `dismissAlert()`, just with different content\. The ntfy polling runs independently of the server connection, so messages work even if the mainline server is offline \(the device just shows the last cached headlines\)\.
+This is architecturally identical to `DoorbellLogic::onAlert()` → `dismissAlert()`, just with different content\. The ntfy polling runs independently of the server connection, so messages work even if the mainline server is offline (the device just shows the last cached headlines)\.
 ### Headline storage
-* Board 3 \(8 MB PSRAM\): store all ~120 headline bitmaps in PSRAM\. At 800px × 16px × 1 bit = 1\.6 KB each → ~192 KB total\. Trivial\.
+* Board 3 (8 MB PSRAM): store all ~120 headline bitmaps in PSRAM\. At 800px × 16px × 1 bit = 1\.6 KB each → ~192 KB total\. Trivial\.
-* Boards 1/2 \(PSRAM TBD\): at 320px × 16px = 640 bytes each → ~77 KB for 120 headlines\. Fits if PSRAM is present\. Without PSRAM, keep ~20 headlines in a ring buffer \(~13 KB\)\.
+* Boards 1/2 (PSRAM TBD): at 320px × 16px = 640 bytes each → ~77 KB for 120 headlines\. Fits if PSRAM is present\. Without PSRAM, keep ~20 headlines in a ring buffer (~13 KB)\.
-### Gradient coloring \(local\)
+### Gradient coloring (local)
 The 12\-step ANSI gradient in mainline\.py maps to 12 RGB565 values:
 ```warp-runnable-command
 const uint16_t GRADIENT[] = {
@@ -68,8 +68,8 @@ mainline.py          (existing, unchanged)
 serve.py             (new — HTTP server, imports mainline rendering functions)
 klubhaus-doorbell-hardware.md  (existing)
 ```
-`serve.py` imports the rendering functions from mainline\.py \(after refactoring them into importable form — they're currently top\-level but not wrapped in `if __name__`\)\.
+`serve.py` imports the rendering functions from mainline\.py (after refactoring them into importable form — they're currently top\-level but not wrapped in `if __name__`)\.
-### klubhaus\-doorbell repo \(or mainline repo under firmware/\)
+### klubhaus\-doorbell repo (or mainline repo under firmware/)
 ```warp-runnable-command
 boards/esp32-mainline/
 ├── esp32-mainline.ino    Main sketch
@@ -79,31 +79,31 @@ boards/esp32-mainline/
 ├── HeadlineStore.h/.cpp  Bitmap ring buffer in PSRAM
 └── NtfyPoller.h/.cpp     ntfy.sh polling (extracted from DoorbellLogic pattern)
 ```
-The display driver is reused from the target board \(e\.g\. `DisplayDriverGFX` for Board 3\)\. `MainlineLogic` replaces `DoorbellLogic` as the state machine but follows the same patterns\.
+The display driver is reused from the target board (e\.g\. `DisplayDriverGFX` for Board 3)\. `MainlineLogic` replaces `DoorbellLogic` as the state machine but follows the same patterns\.
 ## Branch strategy recommendation
 The work spans two repos and has clear dependency ordering\.
-### Phase 1 — Finish current branch \(mainline repo\)
+### Phase 1 — Finish current branch (mainline repo)
-**Branch:** `feat/arduino` \(current\)
+**Branch:** `feat/arduino` (current)
 **Content:** Hardware spec doc\. Already done\.
 **Action:** Merge to main when ready\.
-### Phase 2 — Server renderer \(mainline repo\)
+### Phase 2 — Server renderer (mainline repo)
-**Branch:** `feat/renderer` \(branch from main after Phase 1 merges\)
+**Branch:** `feat/renderer` (branch from main after Phase 1 merges)
 **Content:**
-* Refactor mainline\.py rendering functions to be importable \(extract from `__main__` guard\)
+* Refactor mainline\.py rendering functions to be importable (extract from `__main__` guard)
 * `serve.py` — HTTP server with `/api/headlines`, `/api/config`, `/api/health`
-* Bitmap packing utility \(1\-bit row\-major\)
+* Bitmap packing utility (1\-bit row\-major)
-**Why a separate branch:** This changes mainline\.py's structure \(refactoring for imports\) and adds a new entry point\. It's a self\-contained, testable unit — you can verify the API with `curl` before touching any Arduino code\.
+**Why a separate branch:** This changes mainline\.py's structure (refactoring for imports) and adds a new entry point\. It's a self\-contained, testable unit — you can verify the API with `curl` before touching any Arduino code\.
-### Phase 3 — ESP32 client \(klubhaus\-doorbell repo, or mainline repo\)
+### Phase 3 — ESP32 client (klubhaus\-doorbell repo, or mainline repo)
 **Branch:** `feat/mainline-client` in whichever repo hosts it
 **Content:**
 * `MainlineLogic` state machine
 * `HeadlineStore` bitmap buffer
 * `NtfyPoller` for `klubhaus_terminal_mainline`
 * Board\-specific sketch for the target board
-**Depends on:** Phase 2 \(needs a running server to test against\)
+**Depends on:** Phase 2 (needs a running server to test against)
 **Repo decision:** If you have push access to klubhaus\-doorbell, it fits naturally as a new board target alongside the existing doorbell sketches — it reuses `NetManager`, `IDisplayDriver`, and the vendored display libraries\. If not, put it under `mainline/firmware/` and vendor the shared KlubhausCore library\.
 ### Merge order
-1. `feat/arduino` → main \(hardware spec\)
+1. `feat/arduino` → main (hardware spec)
-2. `feat/renderer` → main \(server\)
+2. `feat/renderer` → main (server)
-3. `feat/mainline-client` → main in whichever repo \(ESP32 client\)
+3. `feat/mainline-client` → main in whichever repo (ESP32 client)
 Each phase is independently testable and doesn't block the other until Phase 3 needs a running server\.

engine/render.py

@@ -31,6 +31,22 @@ GRAD_COLS = [
     "\033[2;38;5;235m",  # near black
 ]
 
+# Complementary sweep for queue messages (opposite hue family from ticker greens)
+MSG_GRAD_COLS = [
+    "\033[1;38;5;231m",  # white
+    "\033[1;38;5;225m",  # pale pink-white
+    "\033[38;5;219m",    # bright pink
+    "\033[38;5;213m",    # hot pink
+    "\033[38;5;207m",    # magenta
+    "\033[38;5;201m",    # bright magenta
+    "\033[38;5;165m",    # orchid-red
+    "\033[38;5;161m",    # ruby-magenta
+    "\033[38;5;125m",    # dark magenta
+    "\033[38;5;89m",     # deep maroon-magenta
+    "\033[2;38;5;89m",   # dim deep maroon-magenta
+    "\033[2;38;5;235m",  # near black
+]
+
 # ─── FONT LOADING ─────────────────────────────────────────
 _FONT_OBJ = None
 _FONT_CACHE = {}
@@ -132,9 +148,10 @@ def big_wrap(text, max_w, fnt=None):
     return out
 
 
-def lr_gradient(rows, offset=0.0):
+def lr_gradient(rows, offset=0.0, grad_cols=None):
     """Color each non-space block character with a shifting left-to-right gradient."""
-    n = len(GRAD_COLS)
+    cols = grad_cols or GRAD_COLS
+    n = len(cols)
     max_x = max((len(r.rstrip()) for r in rows if r.strip()), default=1)
     out = []
     for row in rows:
@@ -148,11 +165,16 @@ def lr_gradient(rows, offset=0.0):
             else:
                 shifted = (x / max(max_x - 1, 1) + offset) % 1.0
                 idx = min(round(shifted * (n - 1)), n - 1)
-                buf.append(f"{GRAD_COLS[idx]}{ch}\033[0m")
+                buf.append(f"{cols[idx]}{ch}{RST}")
         out.append("".join(buf))
     return out
 
 
+def lr_gradient_opposite(rows, offset=0.0):
+    """Complementary (opposite wheel) gradient used for queue message panels."""
+    return lr_gradient(rows, offset, MSG_GRAD_COLS)
+
+
 # ─── HEADLINE BLOCK ASSEMBLY ─────────────────────────────
 def make_block(title, src, ts, w):
     """Render a headline into a content block with color."""

engine/scroll.py

@@ -11,37 +11,9 @@ from datetime import datetime
 
 from engine import config
 from engine.terminal import RST, W_COOL, CLR, tw, th
-from engine.render import big_wrap, lr_gradient, make_block
+from engine.render import big_wrap, lr_gradient, lr_gradient_opposite, make_block
 from engine.effects import noise, glitch_bar, fade_line, vis_trunc, next_headline, firehose_line
 
-_ANSI_LINE_RE = re.compile(r"^(\033\[[0-9;]*m)(.*)(\033\[0m)$")
-FIREHOSE_BG = ""
-
-
-def _overlay_segments(ansi_line, bg=""):
-    """Return (1-indexed column, colored chunk) for non-space runs in a line."""
-    m = _ANSI_LINE_RE.match(ansi_line)
-    if m:
-        color, text, _ = m.groups()
-    else:
-        color, text = "", ansi_line
-    segs = []
-    i = 0
-    while i < len(text):
-        if text[i] == " ":
-            i += 1
-            continue
-        j = i + 1
-        while j < len(text) and text[j] != " ":
-            j += 1
-        chunk = text[i:j]
-        if bg or color:
-            segs.append((i + 1, f"{bg}{color}{chunk}{RST}"))
-        else:
-            segs.append((i + 1, chunk))
-        i = j
-    return segs
-
 
 def stream(items, ntfy_poller, mic_monitor):
     """Main render loop with four layers: message, ticker, scroll motion, firehose."""
@@ -56,25 +28,21 @@ def stream(items, ntfy_poller, mic_monitor):
 
     w, h = tw(), th()
     fh = config.FIREHOSE_H if config.FIREHOSE else 0
-    ticker_view_h = h              # ticker uses full viewport; firehose overlays it
+    ticker_view_h = h - fh         # reserve fixed firehose strip at bottom
     GAP = 3            # blank rows between headlines
     scroll_step_interval = config.SCROLL_DUR / (ticker_view_h + 15) * 2
 
     # Taxonomy:
-    # - message: ntfy interrupt panel at top
+    # - message: centered ntfy overlay panel
     # - ticker: large headline text content
     # - scroll: upward camera motion applied to ticker content
-    # - firehose: carriage-return style overlay drifting above ticker
+    # - firehose: fixed carriage-return style strip pinned at bottom
     # Active ticker blocks: (content_rows, color, canvas_y, meta_idx)
     active = []
     scroll_cam = 0     # viewport top in virtual canvas coords
     ticker_next_y = ticker_view_h  # canvas-y where next block starts (off-screen bottom)
     noise_cache = {}
     scroll_motion_accum = 0.0
-    firehose_top = h - fh
-    firehose_rows = [firehose_line(items, w) for _ in range(fh)] if fh > 0 else []
-    firehose_accum = 0.0
-    firehose_interval = max(config.FRAME_DT, scroll_step_interval * 1.3)
 
     def _noise_at(cy):
         if cy not in noise_cache:
@@ -90,16 +58,17 @@ def stream(items, ntfy_poller, mic_monitor):
         t0 = time.monotonic()
         w, h = tw(), th()
         fh = config.FIREHOSE_H if config.FIREHOSE else 0
-        ticker_view_h = h
+        ticker_view_h = h - fh
 
         # ── Check for ntfy message ────────────────────────
-        msg_h = 0  # rows consumed by message zone at top
+        msg_h = 0
+        msg_overlay = []
         msg = ntfy_poller.get_active_message()
 
         buf = []
         if msg is not None:
             m_title, m_body, m_ts = msg
-            # ── Message zone: pinned to top, scroll continues below ──
+            # ── Message overlay: centered in the viewport ──
             display_text = m_body or m_title or "(empty)"
             display_text = re.sub(r"\s+", " ", display_text.upper())
             cache_key = (display_text, w)
@@ -108,15 +77,17 @@ def stream(items, ntfy_poller, mic_monitor):
                 _msg_cache = (cache_key, msg_rows)
             else:
                 msg_rows = _msg_cache[1]
-            msg_rows = lr_gradient(msg_rows, (time.monotonic() * config.GRAD_SPEED) % 1.0)
+            msg_rows = lr_gradient_opposite(msg_rows, (time.monotonic() * config.GRAD_SPEED) % 1.0)
             # Layout: rendered text + meta + border
             elapsed_s = int(time.monotonic() - m_ts)
             remaining = max(0, config.MESSAGE_DISPLAY_SECS - elapsed_s)
             ts_str = datetime.now().strftime("%H:%M:%S")
+            panel_h = len(msg_rows) + 2  # meta + border
+            panel_top = max(0, (h - panel_h) // 2)
             row_idx = 0
             for mr in msg_rows:
                 ln = vis_trunc(mr, w)
-                buf.append(f"\033[{row_idx+1};1H  {ln}{RST}\033[K")
+                msg_overlay.append(f"\033[{panel_top + row_idx + 1};1H  {ln}{RST}\033[K")
                 row_idx += 1
             # Meta line: title (if distinct) + source + countdown
             meta_parts = []
@@ -124,15 +95,13 @@ def stream(items, ntfy_poller, mic_monitor):
                 meta_parts.append(m_title)
             meta_parts.append(f"ntfy \u00b7 {ts_str} \u00b7 {remaining}s")
             meta = "  " + " \u00b7 ".join(meta_parts) if len(meta_parts) > 1 else "  " + meta_parts[0]
-            buf.append(f"\033[{row_idx+1};1H{MSG_META}{meta}{RST}\033[K")
+            msg_overlay.append(f"\033[{panel_top + row_idx + 1};1H{MSG_META}{meta}{RST}\033[K")
             row_idx += 1
-            # Border — constant boundary between message and scroll
+            # Border — constant boundary under message panel
             bar = "\u2500" * (w - 4)
-            buf.append(f"\033[{row_idx+1};1H  {MSG_BORDER}{bar}{RST}\033[K")
+            msg_overlay.append(f"\033[{panel_top + row_idx + 1};1H  {MSG_BORDER}{bar}{RST}\033[K")
-            row_idx += 1
-            msg_h = row_idx
 
-        # Effective ticker draw height: below message
+        # Ticker draws above the fixed firehose strip; message is a centered overlay.
         ticker_h = ticker_view_h - msg_h
 
         # ── Ticker content + scroll motion (always runs) ──
@@ -156,28 +125,13 @@ def stream(items, ntfy_poller, mic_monitor):
                 if k < scroll_cam:
                     del noise_cache[k]
 
-        # Firehose overlay drift (slower than main ticker)
+        # Draw ticker zone (above fixed firehose strip)
-        if config.FIREHOSE and fh > 0:
-            while len(firehose_rows) < fh:
-                firehose_rows.append(firehose_line(items, w))
-            if len(firehose_rows) > fh:
-                firehose_rows = firehose_rows[-fh:]
-            firehose_accum += config.FRAME_DT
-            while firehose_accum >= firehose_interval:
-                firehose_accum -= firehose_interval
-                firehose_top -= 1
-                firehose_rows.pop(0)
-                firehose_rows.append(firehose_line(items, w))
-            if firehose_top + fh < 0:
-                firehose_top = h - fh
-
-        # Draw ticker zone (below message zone)
         top_zone = max(1, int(ticker_h * 0.25))
         bot_zone = max(1, int(ticker_h * 0.10))
         grad_offset = (time.monotonic() * config.GRAD_SPEED) % 1.0
         ticker_buf_start = len(buf)  # track where ticker rows start in buf
         for r in range(ticker_h):
-            scr_row = msg_h + r + 1  # 1-indexed ANSI screen row
+            scr_row = r + 1  # 1-indexed ANSI screen row
             cy = scroll_cam + r
             top_f = min(1.0, r / top_zone) if top_zone > 0 else 1.0
             bot_f = min(1.0, (ticker_h - 1 - r) / bot_zone) if bot_zone > 0 else 1.0
@@ -219,16 +173,16 @@ def stream(items, ntfy_poller, mic_monitor):
         if random.random() < glitch_prob and ticker_buf_len > 0:
             for _ in range(min(n_hits, ticker_buf_len)):
                 gi = random.randint(0, ticker_buf_len - 1)
-                scr_row = msg_h + gi + 1
+                scr_row = gi + 1
                 buf[ticker_buf_start + gi] = f"\033[{scr_row};1H{glitch_bar(w)}"
 
         if config.FIREHOSE and fh > 0:
-            for fr, fline in enumerate(firehose_rows):
+            for fr in range(fh):
-                scr_row = firehose_top + fr + 1
+                scr_row = h - fh + fr + 1
-                if scr_row <= msg_h or scr_row > h:
+                fline = firehose_line(items, w)
-                    continue
+                buf.append(f"\033[{scr_row};1H{fline}\033[K")
-                for col, chunk in _overlay_segments(fline, FIREHOSE_BG):
+        if msg_overlay:
-                    buf.append(f"\033[{scr_row};{col}H{chunk}")
+            buf.extend(msg_overlay)
 
         sys.stdout.buffer.write("".join(buf).encode())
         sys.stdout.flush()