feat: Implement OTF to terminal half-block rendering pipeline including font loading, text rasterization, word-wrap, and gradient coloring.

Reviewed-on: #7

Mainline Renderer + ntfy Message Queue for ESP32.md

@@ -0,0 +1,109 @@
+# Mainline Renderer \+ ntfy Message Queue for ESP32
+## Problem
+mainline\.py does heavy work unsuitable for ESP32: 25\+ HTTPS/TLS RSS feeds, OTF font rasterization via Pillow, Google Translate API calls, and complex text layout\. Simultaneously, messages arriving on `ntfy.sh/klubhaus_terminal_mainline` need to interrupt the news ticker on the same device\.
+## Architecture: Server \+ Thin Client
+Split the system into two halves that are designed together\.
+**Server \(mainline\.py `--serve` mode, runs on any always\-on machine\)**
+* Reuses existing feed fetching, caching, content filtering, translation, and Pillow font rendering pipeline — no duplication\.
+* Pre\-renders each headline into a 1\-bit bitmap strip \(the OTF→half\-block pipeline already produces this as an intermediate step in `_render_line()`\)\.
+* Exposes a lightweight HTTP API the ESP32 polls\.
+**ESP32 thin client \(Arduino sketch\)**
+* Polls the mainline server for pre\-rendered headline bitmaps over plain HTTP \(no TLS needed if on the same LAN\)\.
+* Polls `ntfy.sh/klubhaus_terminal_mainline` directly for messages, reusing the proven `NetManager::httpGet()` \+ JSON parsing pattern from DoorbellLogic \(`DoorbellLogic.cpp:155-192`\)\.
+* Manages scrolling, gradient coloring, and glitch effects locally \(cheap per\-frame GPU work\)\.
+* When an ntfy message arrives, the scroll is paused and the message takes over the display — same interrupt pattern as the doorbell's ALERT→DASHBOARD flow\.
+## Server API \(mainline repo\)
+New file: `serve.py` \(or `--serve` mode in mainline\.py\)\.
+Endpoints:
+* `GET /api/headlines` — returns JSON array of headline metadata: `[{"id": 0, "src": "Nature", "ts": "14:30", "width": 280, "height": 16, "bitmap": "<base64 1-bit packed>"}]`\. Bitmaps are 1\-bit\-per\-pixel, row\-major, packed 8px/byte\. The ESP32 applies gradient color locally\.
+* `GET /api/config` — returns `{"count": 120, "version": "...", "mode": "news"}` so the ESP32 knows what it's getting\.
+* `GET /api/health` — `{"ok": true, "last_fetch": "...", "headline_count": 120}`
+The server renders at a configurable target width \(e\.g\. 800px for Board 3, 320px for Boards 1/2\) via a `--width` flag or query parameter\. Height is fixed per headline by the font size\.
+The server refreshes feeds on a timer \(reusing `_SCROLL_DUR` cadence or a longer interval\), re\-renders, and serves the latest set\. The ESP32 polls `/api/headlines` periodically \(e\.g\. every 60s\) and swaps in the new set\.
+## Render pipeline \(server side\)
+The existing `_render_line()` in mainline\.py already does:
+1. `ImageFont.truetype()` → `ImageDraw.text()` → grayscale `Image`
+2. Resize to target height
+3. Threshold to 1\-bit \(the `thr = 80` step\)
+For the server, we stop at step 3 and pack the 1\-bit data into bytes instead of converting to half\-block Unicode\. This is the exact same pipeline, just with a different output format\. The `_big_wrap()` and `_lr_gradient()` logic stays on the server for layout; gradient *coloring* moves to the ESP32 \(it's just an index lookup per pixel column\)\.
+## ESP32 client
+### State machine
+```warp-runnable-command
+BOOT → SCROLL ⇄ MESSAGE
+              ↘ OFF (inactivity timeout)
+```
+* **BOOT** — WiFi connect, initial headline fetch from server\.
+* **SCROLL** — Vertical scroll through pre\-rendered headlines with local gradient \+ glitch\. Polls server for new headlines periodically\. Polls ntfy every 15s\.
+* **MESSAGE** — ntfy message arrived\. Scroll paused, message displayed\. Auto\-dismiss after timeout or touch\-dismiss\. Returns to SCROLL\.
+* **OFF** — Backlight off after inactivity \(polling continues in background\)\.
+### ntfy integration
+The ESP32 polls `https://ntfy.sh/klubhaus_terminal_mainline/json?since=20s&poll=1` on the same 15s interval as the doorbell polls its topics\. When a message event arrives:
+1. Parse JSON: `{"event": "message", "title": "...", "message": "..."}`
+2. Save current scroll position\.
+3. Transition to MESSAGE state\.
+4. Render message text using the display library's built\-in fonts \(messages are short, no custom font needed\)\.
+5. After `MESSAGE_TIMEOUT_MS` \(e\.g\. 30s\) or touch, restore scroll position and resume\.
+This is architecturally identical to `DoorbellLogic::onAlert()` → `dismissAlert()`, just with different content\. The ntfy polling runs independently of the server connection, so messages work even if the mainline server is offline \(the device just shows the last cached headlines\)\.
+### Headline storage
+* Board 3 \(8 MB PSRAM\): store all ~120 headline bitmaps in PSRAM\. At 800px × 16px × 1 bit = 1\.6 KB each → ~192 KB total\. Trivial\.
+* Boards 1/2 \(PSRAM TBD\): at 320px × 16px = 640 bytes each → ~77 KB for 120 headlines\. Fits if PSRAM is present\. Without PSRAM, keep ~20 headlines in a ring buffer \(~13 KB\)\.
+### Gradient coloring \(local\)
+The 12\-step ANSI gradient in mainline\.py maps to 12 RGB565 values:
+```warp-runnable-command
+const uint16_t GRADIENT[] = {
+    0xFFFF, // white
+    0xC7FF, // pale cyan
+    0x07F9, // bright cyan
+    0x07C0, // bright lime
+    ...
+    0x0120, // deep green
+    0x18C3, // near black
+};
+```
+For each pixel column `x` in the bitmap, the ESP32 picks `GRADIENT[x * 12 / width]` if the bit is set, else background color\. This is a per\-pixel multiply \+ table lookup — fast\.
+## File layout
+### mainline repo
+```warp-runnable-command
+mainline.py          (existing, unchanged)
+serve.py             (new — HTTP server, imports mainline rendering functions)
+klubhaus-doorbell-hardware.md  (existing)
+```
+`serve.py` imports the rendering functions from mainline\.py \(after refactoring them into importable form — they're currently top\-level but not wrapped in `if __name__`\)\.
+### klubhaus\-doorbell repo \(or mainline repo under firmware/\)
+```warp-runnable-command
+boards/esp32-mainline/
+├── esp32-mainline.ino    Main sketch
+├── board_config.h        Display/pin config (copy from target board)
+├── secrets.h             WiFi creds + server URL
+├── MainlineLogic.h/.cpp  State machine (replaces DoorbellLogic)
+├── HeadlineStore.h/.cpp  Bitmap ring buffer in PSRAM
+└── NtfyPoller.h/.cpp     ntfy.sh polling (extracted from DoorbellLogic pattern)
+```
+The display driver is reused from the target board \(e\.g\. `DisplayDriverGFX` for Board 3\)\. `MainlineLogic` replaces `DoorbellLogic` as the state machine but follows the same patterns\.
+## Branch strategy recommendation
+The work spans two repos and has clear dependency ordering\.
+### Phase 1 — Finish current branch \(mainline repo\)
+**Branch:** `feat/arduino` \(current\)
+**Content:** Hardware spec doc\. Already done\.
+**Action:** Merge to main when ready\.
+### Phase 2 — Server renderer \(mainline repo\)
+**Branch:** `feat/renderer` \(branch from main after Phase 1 merges\)
+**Content:**
+* Refactor mainline\.py rendering functions to be importable \(extract from `__main__` guard\)
+* `serve.py` — HTTP server with `/api/headlines`, `/api/config`, `/api/health`
+* Bitmap packing utility \(1\-bit row\-major\)
+**Why a separate branch:** This changes mainline\.py's structure \(refactoring for imports\) and adds a new entry point\. It's a self\-contained, testable unit — you can verify the API with `curl` before touching any Arduino code\.
+### Phase 3 — ESP32 client \(klubhaus\-doorbell repo, or mainline repo\)
+**Branch:** `feat/mainline-client` in whichever repo hosts it
+**Content:**
+* `MainlineLogic` state machine
+* `HeadlineStore` bitmap buffer
+* `NtfyPoller` for `klubhaus_terminal_mainline`
+* Board\-specific sketch for the target board
+**Depends on:** Phase 2 \(needs a running server to test against\)
+**Repo decision:** If you have push access to klubhaus\-doorbell, it fits naturally as a new board target alongside the existing doorbell sketches — it reuses `NetManager`, `IDisplayDriver`, and the vendored display libraries\. If not, put it under `mainline/firmware/` and vendor the shared KlubhausCore library\.
+### Merge order
+1. `feat/arduino` → main \(hardware spec\)
+2. `feat/renderer` → main \(server\)
+3. `feat/mainline-client` → main in whichever repo \(ESP32 client\)
+Each phase is independently testable and doesn't block the other until Phase 3 needs a running server\.

Refactor mainline.md

@@ -0,0 +1,178 @@
+# Refactor mainline\.py into modular package
+## Problem
+`mainline.py` is a single 1085\-line file with ~10 interleaved concerns\. This prevents:
+* Reusing the ntfy doorbell interrupt in other visualizers
+* Importing the render pipeline from `serve.py` \(future ESP32 HTTP server\)
+* Testing any concern in isolation
+* Porting individual layers to Rust independently
+## Target structure
+```warp-runnable-command
+mainline.py              # thin entrypoint: venv bootstrap → engine.app.main()
+engine/
+  __init__.py
+  config.py              # constants, CLI flags, glyph tables
+  sources.py             # FEEDS, POETRY_SOURCES, SOURCE_LANGS, _LOCATION_LANGS
+  terminal.py            # ANSI codes, tw/th, type_out, slow_print, boot_ln
+  filter.py              # HTML stripping, content filter (_SKIP_RE)
+  translate.py           # Google Translate wrapper + location→language detection
+  render.py              # OTF font loading, _render_line, _big_wrap, _lr_gradient, _make_block
+  effects.py             # noise, glitch_bar, _fade_line, _vis_trunc, _firehose_line, _next_headline
+  fetch.py               # RSS/Gutenberg fetching, cache load/save
+  ntfy.py                # NtfyPoller class — standalone, zero internal deps
+  mic.py                 # MicMonitor class — standalone
+  scroll.py              # stream() frame loop + message rendering
+  app.py                 # main(), TITLE art, boot sequence, signal handler
+```
+The package is named `engine/` to avoid a naming conflict with the `mainline.py` entrypoint\.
+## Module dependency graph
+```warp-runnable-command
+config      ← (nothing)
+sources     ← (nothing)
+terminal    ← (nothing)
+filter      ← (nothing)
+translate   ← sources
+render      ← config, terminal, sources
+effects     ← config, terminal, sources
+fetch       ← config, sources, filter, terminal
+ntfy        ← (nothing — stdlib only, fully standalone)
+mic         ← (nothing — sounddevice only)
+scroll      ← config, terminal, render, effects, ntfy, mic
+app         ← everything above
+```
+Critical property: **ntfy\.py and mic\.py have zero internal dependencies**, making ntfy reusable by any visualizer\.
+## Module details
+### mainline\.py \(entrypoint — slimmed down\)
+Keeps only the venv bootstrap \(lines 10\-38\) which must run before any third\-party imports\. After bootstrap, delegates to `engine.app.main()`\.
+### engine/config\.py
+From current mainline\.py:
+* `HEADLINE_LIMIT`, `FEED_TIMEOUT`, `MIC_THRESHOLD_DB` \(lines 55\-57\)
+* `MODE`, `FIREHOSE` CLI flag parsing \(lines 58\-59\)
+* `NTFY_TOPIC`, `NTFY_POLL_INTERVAL`, `MESSAGE_DISPLAY_SECS` \(lines 62\-64\)
+* `_FONT_PATH`, `_FONT_SZ`, `_RENDER_H` \(lines 147\-150\)
+* `_SCROLL_DUR`, `_FRAME_DT`, `FIREHOSE_H` \(lines 505\-507\)
+* `GLITCH`, `KATA` glyph tables \(lines 143\-144\)
+### engine/sources\.py
+Pure data, no logic:
+* `FEEDS` dict \(lines 102\-140\)
+* `POETRY_SOURCES` dict \(lines 67\-80\)
+* `SOURCE_LANGS` dict \(lines 258\-266\)
+* `_LOCATION_LANGS` dict \(lines 269\-289\)
+* `_SCRIPT_FONTS` dict \(lines 153\-165\)
+* `_NO_UPPER` set \(line 167\)
+### engine/terminal\.py
+ANSI primitives and terminal I/O:
+* All ANSI constants: `RST`, `BOLD`, `DIM`, `G_HI`, `G_MID`, `G_LO`, `G_DIM`, `W_COOL`, `W_DIM`, `W_GHOST`, `C_DIM`, `CLR`, `CURSOR_OFF`, `CURSOR_ON` \(lines 83\-99\)
+* `tw()`, `th()` \(lines 223\-234\)
+* `type_out()`, `slow_print()`, `boot_ln()` \(lines 355\-386\)
+### engine/filter\.py
+* `_Strip` HTML parser class \(lines 205\-214\)
+* `strip_tags()` \(lines 217\-220\)
+* `_SKIP_RE` compiled regex \(lines 322\-346\)
+* `_skip()` predicate \(lines 349\-351\)
+### engine/translate\.py
+* `_TRANSLATE_CACHE` \(line 291\)
+* `_detect_location_language()` \(lines 294\-300\) — imports `_LOCATION_LANGS` from sources
+* `_translate_headline()` \(lines 303\-319\)
+### engine/render\.py
+The OTF→terminal pipeline\. This is exactly what `serve.py` will import to produce 1\-bit bitmaps for the ESP32\.
+* `_GRAD_COLS` gradient table \(lines 169\-182\)
+* `_font()`, `_font_for_lang()` with lazy\-load \+ cache \(lines 185\-202\)
+* `_render_line()` — OTF text → half\-block terminal rows \(lines 567\-605\)
+* `_big_wrap()` — word\-wrap \+ render \(lines 608\-636\)
+* `_lr_gradient()` — apply left→right color gradient \(lines 639\-656\)
+* `_make_block()` — composite: translate → render → colorize a headline \(lines 718\-756\)\. Imports from translate, sources\.
+### engine/effects\.py
+Visual effects applied during the frame loop:
+* `noise()` \(lines 237\-245\)
+* `glitch_bar()` \(lines 248\-252\)
+* `_fade_line()` — probabilistic character dissolve \(lines 659\-680\)
+* `_vis_trunc()` — ANSI\-aware width truncation \(lines 683\-701\)
+* `_firehose_line()` \(lines 759\-801\) — imports config\.MODE, sources\.FEEDS/POETRY\_SOURCES
+* `_next_headline()` — pool management \(lines 704\-715\)
+### engine/fetch\.py
+* `fetch_feed()` \(lines 390\-396\)
+* `fetch_all()` \(lines 399\-426\) — imports filter\.\_skip, filter\.strip\_tags, terminal\.boot\_ln
+* `_fetch_gutenberg()` \(lines 429\-456\)
+* `fetch_poetry()` \(lines 459\-472\)
+* `_cache_path()`, `_load_cache()`, `_save_cache()` \(lines 476\-501\)
+### engine/ntfy\.py — standalone, reusable
+Refactored from the current globals \+ thread \(lines 531\-564\) and the message rendering section of `stream()` \(lines 845\-909\) into a class:
+```python
+class NtfyPoller:
+    def __init__(self, topic_url, poll_interval=15, display_secs=30):
+        ...
+    def start(self):
+        """Start background polling thread."""
+    def get_active_message(self):
+        """Return (title, body, timestamp) if a message is active and not expired, else None."""
+    def dismiss(self):
+        """Manually dismiss current message."""
+```
+Dependencies: `urllib.request`, `json`, `threading`, `time` — all stdlib\. No internal imports\.
+Other visualizers use it like:
+```python
+from engine.ntfy import NtfyPoller
+poller = NtfyPoller("https://ntfy.sh/my_topic/json?since=20s&poll=1")
+poller.start()
+# in render loop:
+msg = poller.get_active_message()
+if msg:
+    title, body, ts = msg
+    render_my_message(title, body)  # visualizer-specific
+```
+### engine/mic\.py — standalone
+Refactored from the current globals \(lines 508\-528\) into a class:
+```python
+class MicMonitor:
+    def __init__(self, threshold_db=50):
+        ...
+    def start(self) -> bool:
+        """Start background mic stream. Returns False if unavailable."""
+    def stop(self):
+        ...
+    @property
+    def db(self) -> float:
+        """Current RMS dB level."""
+    @property
+    def excess(self) -> float:
+        """dB above threshold (clamped to 0)."""
+```
+Dependencies: `sounddevice`, `numpy` \(both optional — graceful fallback\)\.
+### engine/scroll\.py
+The `stream()` function \(lines 804\-990\)\. Receives its dependencies via arguments or imports:
+* `stream(items, ntfy_poller, mic_monitor, config)` or similar
+* Message rendering \(lines 855\-909\) stays here since it's terminal\-display\-specific — a different visualizer would render messages differently
+### engine/app\.py
+The orchestrator:
+* `TITLE` ASCII art \(lines 994\-1001\)
+* `main()` \(lines 1004\-1084\): CLI handling, signal setup, boot animation, fetch, wire up ntfy/mic/scroll
+## Execution order
+### Step 1: Create engine/ package skeleton
+Create `engine/__init__.py` and all empty module files\.
+### Step 2: Extract pure data modules \(zero\-dep\)
+Move constants and data dicts into `config.py`, `sources.py`\. These have no logic dependencies\.
+### Step 3: Extract terminal\.py
+Move ANSI codes and terminal I/O helpers\. No internal deps\.
+### Step 4: Extract filter\.py and translate\.py
+Both are small, self\-contained\. translate imports from sources\.
+### Step 5: Extract render\.py
+Font loading \+ the OTF→half\-block pipeline\. Imports from config, terminal, sources\. This is the module `serve.py` will later import\.
+### Step 6: Extract effects\.py
+Visual effects\. Imports from config, terminal, sources\.
+### Step 7: Extract fetch\.py
+Feed/Gutenberg fetching \+ caching\. Imports from config, sources, filter, terminal\.
+### Step 8: Extract ntfy\.py and mic\.py
+Refactor globals\+threads into classes\. Zero internal deps\.
+### Step 9: Extract scroll\.py
+The frame loop\. Last to extract because it depends on everything above\.
+### Step 10: Extract app\.py
+The `main()` function, boot sequence, signal handler\. Wire up all modules\.
+### Step 11: Slim down mainline\.py
+Keep only venv bootstrap \+ `from engine.app import main; main()`\.
+### Step 12: Verify
+Run `python3 mainline.py`, `python3 mainline.py --poetry`, and `python3 mainline.py --firehose` to confirm identical behavior\. No behavioral changes in this refactor\.
+## What this enables
+* **serve\.py** \(future\): `from engine.render import _render_line, _big_wrap` \+ `from engine.fetch import fetch_all` — imports the pipeline directly
+* **Other visualizers**: `from engine.ntfy import NtfyPoller` — doorbell feature with no coupling to mainline's scroll engine
+* **Rust port**: Clear boundaries for what to port first \(ntfy client, render pipeline\) vs what stays in Python \(fetching, caching — the server side\)
+* **Testing**: Each module can be unit\-tested in isolation

engine/__init__.py

@@ -0,0 +1 @@
+# engine — modular internals for mainline

engine/config.py

@@ -0,0 +1,31 @@
+"""
+Configuration constants, CLI flags, and glyph tables.
+"""
+
+import sys
+
+# ─── RUNTIME ──────────────────────────────────────────────
+HEADLINE_LIMIT = 1000
+FEED_TIMEOUT = 10
+MIC_THRESHOLD_DB = 50   # dB above which glitches intensify
+MODE = 'poetry' if '--poetry' in sys.argv or '-p' in sys.argv else 'news'
+FIREHOSE = '--firehose' in sys.argv
+
+# ─── NTFY MESSAGE QUEUE ──────────────────────────────────
+NTFY_TOPIC = "https://ntfy.sh/klubhaus_terminal_mainline/json?since=20s&poll=1"
+NTFY_POLL_INTERVAL = 15   # seconds between polls
+MESSAGE_DISPLAY_SECS = 30 # how long a message holds the screen
+
+# ─── FONT RENDERING ──────────────────────────────────────
+FONT_PATH = "/Users/genejohnson/Documents/CS Bishop Drawn/CSBishopDrawn-Italic.otf"
+FONT_SZ = 60
+RENDER_H = 8              # terminal rows per rendered text line
+
+# ─── SCROLL / FRAME ──────────────────────────────────────
+SCROLL_DUR = 3.75         # seconds per headline
+FRAME_DT = 0.05           # 50ms base frame rate (20 FPS)
+FIREHOSE_H = 12           # firehose zone height (terminal rows)
+
+# ─── GLYPHS ───────────────────────────────────────────────
+GLITCH = "░▒▓█▌▐╌╍╎╏┃┆┇┊┋"
+KATA = "ﾊﾐﾋｰｳｼﾅﾓﾆｻﾜﾂｵﾘｱﾎﾃﾏｹﾒｴｶｷﾑﾕﾗｾﾈｽﾀﾇﾍ"

engine/filter.py

@@ -0,0 +1,60 @@
+"""
+HTML stripping and content filter (sports, vapid, insipid).
+No internal dependencies.
+"""
+
+import re
+from html import unescape
+from html.parser import HTMLParser
+
+
+# ─── HTML STRIPPING ───────────────────────────────────────
+class _Strip(HTMLParser):
+    def __init__(self):
+        super().__init__()
+        self._t = []
+
+    def handle_data(self, d):
+        self._t.append(d)
+
+    def text(self):
+        return "".join(self._t).strip()
+
+
+def strip_tags(html):
+    s = _Strip()
+    s.feed(unescape(html or ""))
+    return s.text()
+
+
+# ─── CONTENT FILTER ───────────────────────────────────────
+_SKIP_RE = re.compile(
+    r'\b(?:'
+    # ── sports ──
+    r'football|soccer|basketball|baseball|softball|tennis|golf|cricket|rugby|'
+    r'hockey|lacrosse|volleyball|badminton|'
+    r'nba|nfl|nhl|mlb|mls|fifa|uefa|'
+    r'premier league|champions league|la liga|serie a|bundesliga|'
+    r'world cup|super bowl|world series|stanley cup|'
+    r'playoff|playoffs|touchdown|goalkeeper|striker|quarterback|'
+    r'slam dunk|home run|grand slam|offside|halftime|'
+    r'batting|wicket|innings|'
+    r'formula 1|nascar|motogp|'
+    r'boxing|ufc|mma|'
+    r'marathon|tour de france|'
+    r'transfer window|draft pick|relegation|'
+    # ── vapid / insipid ──
+    r'kardashian|jenner|reality tv|reality show|'
+    r'influencer|viral video|tiktok|instagram|'
+    r'best dressed|worst dressed|red carpet|'
+    r'horoscope|zodiac|gossip|bikini|selfie|'
+    r'you won.t believe|what happened next|'
+    r'celebrity couple|celebrity feud|baby bump'
+    r')\b',
+    re.IGNORECASE
+)
+
+
+def skip(title):
+    """Return True if headline is sports, vapid, or insipid."""
+    return bool(_SKIP_RE.search(title))

engine/render.py

@@ -0,0 +1,191 @@
+"""
+OTF → terminal half-block rendering pipeline.
+Font loading, text rasterization, word-wrap, gradient coloring, headline block assembly.
+Depends on: config, terminal, sources, translate.
+"""
+
+import re
+import random
+
+from PIL import Image, ImageDraw, ImageFont
+
+from engine import config
+from engine.terminal import RST, W_COOL
+from engine.sources import SCRIPT_FONTS, SOURCE_LANGS, NO_UPPER
+from engine.translate import detect_location_language, translate_headline
+
+# ─── GRADIENT ─────────────────────────────────────────────
+# Left → right: white-hot leading edge fades to near-black
+GRAD_COLS = [
+    "\033[1;38;5;231m",  # white
+    "\033[1;38;5;195m",  # pale cyan-white
+    "\033[38;5;123m",    # bright cyan
+    "\033[38;5;118m",    # bright lime
+    "\033[38;5;82m",     # lime
+    "\033[38;5;46m",     # bright green
+    "\033[38;5;40m",     # green
+    "\033[38;5;34m",     # medium green
+    "\033[38;5;28m",     # dark green
+    "\033[38;5;22m",     # deep green
+    "\033[2;38;5;22m",   # dim deep green
+    "\033[2;38;5;235m",  # near black
+]
+
+# ─── FONT LOADING ─────────────────────────────────────────
+_FONT_OBJ = None
+_FONT_CACHE = {}
+
+
+def font():
+    """Lazy-load the primary OTF font."""
+    global _FONT_OBJ
+    if _FONT_OBJ is None:
+        _FONT_OBJ = ImageFont.truetype(config.FONT_PATH, config.FONT_SZ)
+    return _FONT_OBJ
+
+
+def font_for_lang(lang=None):
+    """Get appropriate font for a language."""
+    if lang is None or lang not in SCRIPT_FONTS:
+        return font()
+    if lang not in _FONT_CACHE:
+        try:
+            _FONT_CACHE[lang] = ImageFont.truetype(SCRIPT_FONTS[lang], config.FONT_SZ)
+        except Exception:
+            _FONT_CACHE[lang] = font()
+    return _FONT_CACHE[lang]
+
+
+# ─── RASTERIZATION ────────────────────────────────────────
+def render_line(text, fnt=None):
+    """Render a line of text as terminal rows using OTF font + half-blocks."""
+    if fnt is None:
+        fnt = font()
+    bbox = fnt.getbbox(text)
+    if not bbox or bbox[2] <= bbox[0]:
+        return [""]
+    pad = 4
+    img_w = bbox[2] - bbox[0] + pad * 2
+    img_h = bbox[3] - bbox[1] + pad * 2
+    img = Image.new('L', (img_w, img_h), 0)
+    draw = ImageDraw.Draw(img)
+    draw.text((-bbox[0] + pad, -bbox[1] + pad), text, fill=255, font=fnt)
+    pix_h = config.RENDER_H * 2
+    scale = pix_h / max(img_h, 1)
+    new_w = max(1, int(img_w * scale))
+    img = img.resize((new_w, pix_h), Image.Resampling.LANCZOS)
+    data = img.tobytes()
+    thr = 80
+    rows = []
+    for y in range(0, pix_h, 2):
+        row = []
+        for x in range(new_w):
+            top = data[y * new_w + x] > thr
+            bot = data[(y + 1) * new_w + x] > thr if y + 1 < pix_h else False
+            if top and bot:
+                row.append("█")
+            elif top:
+                row.append("▀")
+            elif bot:
+                row.append("▄")
+            else:
+                row.append(" ")
+        rows.append("".join(row))
+    while rows and not rows[-1].strip():
+        rows.pop()
+    while rows and not rows[0].strip():
+        rows.pop(0)
+    return rows if rows else [""]
+
+
+def big_wrap(text, max_w, fnt=None):
+    """Word-wrap text and render with OTF font."""
+    if fnt is None:
+        fnt = font()
+    words = text.split()
+    lines, cur = [], ""
+    for word in words:
+        test = f"{cur} {word}".strip() if cur else word
+        bbox = fnt.getbbox(test)
+        if bbox:
+            img_h = bbox[3] - bbox[1] + 8
+            pix_h = config.RENDER_H * 2
+            scale = pix_h / max(img_h, 1)
+            term_w = int((bbox[2] - bbox[0] + 8) * scale)
+        else:
+            term_w = 0
+        if term_w > max_w - 4 and cur:
+            lines.append(cur)
+            cur = word
+        else:
+            cur = test
+    if cur:
+        lines.append(cur)
+    out = []
+    for i, ln in enumerate(lines):
+        out.extend(render_line(ln, fnt))
+        if i < len(lines) - 1:
+            out.append("")
+    return out
+
+
+def lr_gradient(rows):
+    """Color each non-space block character with a left-to-right gradient."""
+    n = len(GRAD_COLS)
+    max_x = max((len(r.rstrip()) for r in rows if r.strip()), default=1)
+    out = []
+    for row in rows:
+        if not row.strip():
+            out.append(row)
+            continue
+        buf = []
+        for x, ch in enumerate(row):
+            if ch == ' ':
+                buf.append(' ')
+            else:
+                idx = min(round(x / max(max_x - 1, 1) * (n - 1)), n - 1)
+                buf.append(f"{GRAD_COLS[idx]}{ch}\033[0m")
+        out.append("".join(buf))
+    return out
+
+
+# ─── HEADLINE BLOCK ASSEMBLY ─────────────────────────────
+def make_block(title, src, ts, w):
+    """Render a headline into a content block with color."""
+    target_lang = (SOURCE_LANGS.get(src) or detect_location_language(title)) if config.MODE == 'news' else None
+    lang_font = font_for_lang(target_lang)
+    if target_lang:
+        title = translate_headline(title, target_lang)
+    # Don't uppercase scripts that have no case (CJK, Arabic, etc.)
+    if target_lang and target_lang in NO_UPPER:
+        title_up = re.sub(r"\s+", " ", title)
+    else:
+        title_up = re.sub(r"\s+", " ", title.upper())
+    for old, new in [("\u2019","'"), ("\u2018","'"), ("\u201c",'"'),
+                     ("\u201d",'"'), ("\u2013","-"), ("\u2014","-")]:
+        title_up = title_up.replace(old, new)
+    big_rows = big_wrap(title_up, w - 4, lang_font)
+    big_rows = lr_gradient(big_rows)
+    hc = random.choice([
+        "\033[38;5;46m",    # matrix green
+        "\033[38;5;34m",    # dark green
+        "\033[38;5;82m",    # lime
+        "\033[38;5;48m",    # sea green
+        "\033[38;5;37m",    # teal
+        "\033[38;5;44m",    # cyan
+        "\033[38;5;87m",    # sky
+        "\033[38;5;117m",   # ice blue
+        "\033[38;5;250m",   # cool white
+        "\033[38;5;156m",   # pale green
+        "\033[38;5;120m",   # mint
+        "\033[38;5;80m",    # dark cyan
+        "\033[38;5;108m",   # grey-green
+        "\033[38;5;115m",   # sage
+        "\033[1;38;5;46m",  # bold green
+        "\033[1;38;5;250m", # bold white
+    ])
+    content = ["  " + r for r in big_rows]
+    content.append("")
+    meta = f"\u2591 {src} \u00b7 {ts}"
+    content.append(" " * max(2, w - len(meta) - 2) + meta)
+    return content, hc, len(content) - 1  # (rows, color, meta_row_index)

engine/sources.py

@@ -0,0 +1,115 @@
+"""
+Data sources: feed URLs, poetry sources, language mappings, script fonts.
+Pure data — no logic, no dependencies.
+"""
+
+# ─── RSS FEEDS ────────────────────────────────────────────
+FEEDS = {
+    # Science & Technology
+    "Nature": "https://www.nature.com/nature.rss",
+    "Science Daily": "https://www.sciencedaily.com/rss/all.xml",
+    "Phys.org": "https://phys.org/rss-feed/",
+    "NASA": "https://www.nasa.gov/news-release/feed/",
+    "Ars Technica": "https://feeds.arstechnica.com/arstechnica/index",
+    "New Scientist": "https://www.newscientist.com/section/news/feed/",
+    "Quanta": "https://api.quantamagazine.org/feed/",
+    "BBC Science": "http://feeds.bbci.co.uk/news/science_and_environment/rss.xml",
+    "MIT Tech Review": "https://www.technologyreview.com/feed/",
+    # Economics & Business
+    "BBC Business": "http://feeds.bbci.co.uk/news/business/rss.xml",
+    "MarketWatch": "https://feeds.marketwatch.com/marketwatch/topstories/",
+    "Economist": "https://www.economist.com/finance-and-economics/rss.xml",
+    # World & Politics
+    "BBC World": "http://feeds.bbci.co.uk/news/world/rss.xml",
+    "NPR": "https://feeds.npr.org/1001/rss.xml",
+    "Al Jazeera": "https://www.aljazeera.com/xml/rss/all.xml",
+    "Guardian World": "https://www.theguardian.com/world/rss",
+    "DW": "https://rss.dw.com/rdf/rss-en-all",
+    "France24": "https://www.france24.com/en/rss",
+    "ABC Australia": "https://www.abc.net.au/news/feed/2942460/rss.xml",
+    "Japan Times": "https://www.japantimes.co.jp/feed/",
+    "The Hindu": "https://www.thehindu.com/news/national/feeder/default.rss",
+    "SCMP": "https://www.scmp.com/rss/91/feed",
+    "Der Spiegel": "https://www.spiegel.de/international/index.rss",
+    # Culture & Ideas
+    "Guardian Culture": "https://www.theguardian.com/culture/rss",
+    "Aeon": "https://aeon.co/feed.rss",
+    "Smithsonian": "https://www.smithsonianmag.com/rss/latest_articles/",
+    "The Marginalian": "https://www.themarginalian.org/feed/",
+    "Nautilus": "https://nautil.us/feed/",
+    "Wired": "https://www.wired.com/feed/rss",
+    "The Conversation": "https://theconversation.com/us/articles.atom",
+    "Longreads": "https://longreads.com/feed/",
+    "Literary Hub": "https://lithub.com/feed/",
+    "Atlas Obscura": "https://www.atlasobscura.com/feeds/latest",
+}
+
+# ─── POETRY / LITERATURE ─────────────────────────────────
+# Public domain via Project Gutenberg
+POETRY_SOURCES = {
+    "Whitman":    "https://www.gutenberg.org/cache/epub/1322/pg1322.txt",
+    "Dickinson":  "https://www.gutenberg.org/cache/epub/12242/pg12242.txt",
+    "Whitman II": "https://www.gutenberg.org/cache/epub/8388/pg8388.txt",
+    "Rilke":      "https://www.gutenberg.org/cache/epub/38594/pg38594.txt",
+    "Pound":      "https://www.gutenberg.org/cache/epub/41162/pg41162.txt",
+    "Pound II":   "https://www.gutenberg.org/cache/epub/51992/pg51992.txt",
+    "Eliot":      "https://www.gutenberg.org/cache/epub/1567/pg1567.txt",
+    "Yeats":      "https://www.gutenberg.org/cache/epub/38877/pg38877.txt",
+    "Masters":    "https://www.gutenberg.org/cache/epub/1280/pg1280.txt",
+    "Baudelaire": "https://www.gutenberg.org/cache/epub/36098/pg36098.txt",
+    "Crane":      "https://www.gutenberg.org/cache/epub/40786/pg40786.txt",
+    "Poe":        "https://www.gutenberg.org/cache/epub/10031/pg10031.txt",
+}
+
+# ─── SOURCE → LANGUAGE MAPPING ───────────────────────────
+# Headlines from these outlets render in their cultural home language
+SOURCE_LANGS = {
+    "Der Spiegel":  "de",
+    "DW":           "de",
+    "France24":     "fr",
+    "Japan Times":  "ja",
+    "The Hindu":    "hi",
+    "SCMP":         "zh-cn",
+    "Al Jazeera":   "ar",
+}
+
+# ─── LOCATION → LANGUAGE ─────────────────────────────────
+LOCATION_LANGS = {
+    r'\b(?:china|chinese|beijing|shanghai|hong kong|xi jinping)\b': 'zh-cn',
+    r'\b(?:japan|japanese|tokyo|osaka|kishida)\b': 'ja',
+    r'\b(?:korea|korean|seoul|pyongyang)\b': 'ko',
+    r'\b(?:russia|russian|moscow|kremlin|putin)\b': 'ru',
+    r'\b(?:saudi|dubai|qatar|egypt|cairo|arabic)\b': 'ar',
+    r'\b(?:india|indian|delhi|mumbai|modi)\b': 'hi',
+    r'\b(?:germany|german|berlin|munich|scholz)\b': 'de',
+    r'\b(?:france|french|paris|lyon|macron)\b': 'fr',
+    r'\b(?:spain|spanish|madrid)\b': 'es',
+    r'\b(?:italy|italian|rome|milan|meloni)\b': 'it',
+    r'\b(?:portugal|portuguese|lisbon)\b': 'pt',
+    r'\b(?:brazil|brazilian|são paulo|lula)\b': 'pt',
+    r'\b(?:greece|greek|athens)\b': 'el',
+    r'\b(?:turkey|turkish|istanbul|ankara|erdogan)\b': 'tr',
+    r'\b(?:iran|iranian|tehran)\b': 'fa',
+    r'\b(?:thailand|thai|bangkok)\b': 'th',
+    r'\b(?:vietnam|vietnamese|hanoi)\b': 'vi',
+    r'\b(?:ukraine|ukrainian|kyiv|kiev|zelensky)\b': 'uk',
+    r'\b(?:israel|israeli|jerusalem|tel aviv|netanyahu)\b': 'he',
+}
+
+# ─── NON-LATIN SCRIPT FONTS (macOS) ──────────────────────
+SCRIPT_FONTS = {
+    'zh-cn': '/System/Library/Fonts/STHeiti Medium.ttc',
+    'ja':    '/System/Library/Fonts/ヒラギノ角ゴシック W9.ttc',
+    'ko':    '/System/Library/Fonts/AppleSDGothicNeo.ttc',
+    'ru':    '/System/Library/Fonts/Supplemental/Arial.ttf',
+    'uk':    '/System/Library/Fonts/Supplemental/Arial.ttf',
+    'el':    '/System/Library/Fonts/Supplemental/Arial.ttf',
+    'he':    '/System/Library/Fonts/Supplemental/Arial.ttf',
+    'ar':    '/System/Library/Fonts/GeezaPro.ttc',
+    'fa':    '/System/Library/Fonts/GeezaPro.ttc',
+    'hi':    '/System/Library/Fonts/Kohinoor.ttc',
+    'th':    '/System/Library/Fonts/ThonburiUI.ttc',
+}
+
+# Scripts that have no uppercase
+NO_UPPER = {'zh-cn', 'ja', 'ko', 'ar', 'fa', 'hi', 'th', 'he'}

engine/terminal.py

@@ -0,0 +1,78 @@
+"""
+ANSI escape codes, terminal size helpers, and text output primitives.
+No internal dependencies.
+"""
+
+import os
+import sys
+import random
+import time
+
+# ─── ANSI ─────────────────────────────────────────────────
+RST = "\033[0m"
+BOLD = "\033[1m"
+DIM = "\033[2m"
+# Matrix greens
+G_HI = "\033[38;5;46m"
+G_MID = "\033[38;5;34m"
+G_LO = "\033[38;5;22m"
+G_DIM = "\033[2;38;5;34m"
+# THX-1138 sterile tones
+W_COOL = "\033[38;5;250m"
+W_DIM = "\033[2;38;5;245m"
+W_GHOST = "\033[2;38;5;238m"
+C_DIM = "\033[2;38;5;37m"
+# Terminal control
+CLR = "\033[2J\033[H"
+CURSOR_OFF = "\033[?25l"
+CURSOR_ON = "\033[?25h"
+
+
+# ─── TERMINAL SIZE ────────────────────────────────────────
+def tw():
+    try:
+        return os.get_terminal_size().columns
+    except Exception:
+        return 80
+
+
+def th():
+    try:
+        return os.get_terminal_size().lines
+    except Exception:
+        return 24
+
+
+# ─── TEXT OUTPUT ──────────────────────────────────────────
+def type_out(text, color=G_HI):
+    i = 0
+    while i < len(text):
+        if random.random() < 0.3:
+            b = random.randint(2, 5)
+            sys.stdout.write(f"{color}{text[i:i+b]}{RST}")
+            i += b
+        else:
+            sys.stdout.write(f"{color}{text[i]}{RST}")
+            i += 1
+        sys.stdout.flush()
+        time.sleep(random.uniform(0.004, 0.018))
+
+
+def slow_print(text, color=G_DIM, delay=0.015):
+    for ch in text:
+        sys.stdout.write(f"{color}{ch}{RST}")
+        sys.stdout.flush()
+        time.sleep(delay)
+
+
+def boot_ln(label, status, ok=True):
+    dots = max(3, min(30, tw() - len(label) - len(status) - 8))
+    sys.stdout.write(f"  {G_DIM}>{RST} {W_DIM}{label} ")
+    sys.stdout.flush()
+    for _ in range(dots):
+        sys.stdout.write(f"{G_LO}.")
+        sys.stdout.flush()
+        time.sleep(random.uniform(0.006, 0.025))
+    c = G_MID if ok else "\033[2;38;5;196m"
+    print(f" {c}{status}{RST}")
+    time.sleep(random.uniform(0.02, 0.1))

engine/translate.py

@@ -0,0 +1,41 @@
+"""
+Google Translate wrapper and location→language detection.
+Depends on: sources (for LOCATION_LANGS).
+"""
+
+import re
+import json
+import urllib.request
+import urllib.parse
+
+from engine.sources import LOCATION_LANGS
+
+_TRANSLATE_CACHE = {}
+
+
+def detect_location_language(title):
+    """Detect if headline mentions a location, return target language."""
+    title_lower = title.lower()
+    for pattern, lang in LOCATION_LANGS.items():
+        if re.search(pattern, title_lower):
+            return lang
+    return None
+
+
+def translate_headline(title, target_lang):
+    """Translate headline via Google Translate API (zero dependencies)."""
+    key = (title, target_lang)
+    if key in _TRANSLATE_CACHE:
+        return _TRANSLATE_CACHE[key]
+    try:
+        q = urllib.parse.quote(title)
+        url = ("https://translate.googleapis.com/translate_a/single"
+               f"?client=gtx&sl=en&tl={target_lang}&dt=t&q={q}")
+        req = urllib.request.Request(url, headers={"User-Agent": "mainline/0.1"})
+        resp = urllib.request.urlopen(req, timeout=5)
+        data = json.loads(resp.read())
+        result = "".join(p[0] for p in data[0] if p[0]) or title
+    except Exception:
+        result = title
+    _TRANSLATE_CACHE[key] = result
+    return result

klubhaus-doorbell-hardware.md

@@ -139,25 +139,27 @@ Things that are plausibly relevant but entirely absent from the codebase:
 
 ## Questions for board owner
 
-I'm looking at porting [mainline.py](mainline.py) — a scrolling terminal news/poetry stream with OTF-font rendering, RSS feeds, ANSI gradients, and glitch effects — to run on one of these boards. To figure out the right approach I need a few things only you can answer:
+I'm looking at displaying [mainline.py](mainline.py) — a scrolling news/poetry consciousness stream — on one of these boards. The plan ("Mainline Renderer + ntfy Message Queue for ESP32") uses a **server + thin client** architecture: a Python server pre-renders headlines and serves them via HTTP; the ESP32 just displays pre-rendered bitmaps, applies gradient/glitch effects locally, and polls `ntfy.sh/klubhaus_terminal_mainline` for messages that interrupt the news scroll.
 
-### 1. Which board should I target?
+To build this I need the following from you:
 
-The three boards have very different constraints:
+### 1. Which board?
+
+With a renderer server doing the heavy lifting, all three boards are viable — but the experience differs:
 
 | | Board 1 (2.8″) | Board 2 (4.0″) | Board 3 (4.3″) |
 |---|---|---|---|
 | Resolution | 320 × 240 | 320 × 480 | 800 × 480 |
-| Display bus | SPI (40 MHz) | SPI (40 MHz) | RGB parallel (14 MHz pclk) |
+| Headline buffer (120 items) | ~77 KB | ~77 KB | ~192 KB |
-| Flash | 4 MB | 4 MB | 16 MB |
+| Firehose mode | no (too narrow) | no (SPI too slow) | yes |
-| PSRAM | unknown | unknown | 8 MB |
+| Smooth scroll at 20 FPS | yes (partial updates) | tight (partial updates mandatory) | yes (framebuffer) |
-| Full-screen redraw | ~60 ms+ | ~120 ms+ | near-instant (framebuffer) |
+| Flash for caching | 4 MB (tight) | 4 MB (tight) | 16 MB (9 MB FAT partition) |
 
-Board 3 is the only one with enough RAM and display bandwidth for smooth scrolling with many headlines buffered. Boards 1 & 2 would need aggressive feature cuts. **Which board do you want this on?**
+Board 3 is the most capable. Boards 1 & 2 work but lose firehose mode and need careful partial-update rendering. **Which board do you want this on?**
 
-### 2. PSRAM on your ESP32-32E boards
+### 2. PSRAM on Boards 1 & 2
 
-The build flags say `-DBOARD_HAS_PSRAM` but I can't tell the capacity. Can you check? Easiest way:
+The build flags say `-DBOARD_HAS_PSRAM` but I can't confirm the capacity. Can you check?
 
 ```
 // Add to setup() temporarily:
@@ -165,27 +167,124 @@ Serial.printf("PSRAM size: %d bytes\n", ESP.getPsramSize());
 Serial.printf("Free PSRAM: %d bytes\n", ESP.getFreePsram());
 ```
 
-If PSRAM is 0 on Boards 1 or 2, those boards can only hold a handful of headlines in 520 KB SRAM (WiFi + TLS eat most of it).
+If PSRAM is 0, the board can only buffer ~20 headlines in a ring buffer instead of the full ~120 set. (Board 3's 8 MB PSRAM is confirmed — this only matters if you pick Board 1 or 2.)
 
-### 3. Feature priorities
+### 3. Network & server hosting
 
-mainline.py does a lot of things that don't map directly to an ESP32 + TFT. Which of these matter to you?
+The renderer server (`serve.py`) needs Python 3 + Pillow, internet access (for RSS feeds), and network access to the ESP32.
 
-- **RSS headline scrolling** — the core experience. How many feeds? All ~25, or a curated subset?
+- **Where will the server run?** Raspberry Pi, NAS, always-on desktop, cloud VM?
-- **OTF font rendering** — mainline uses Pillow to rasterize a custom `.otf` font into half-block characters. On ESP32 this would become either bitmap fonts or a pre-rendered glyph atlas baked into flash. Is the specific font important, or is the aesthetic (large, stylized text) what matters?
+- **Same LAN as the ESP32?** If yes, the ESP32 can use plain HTTP (no TLS overhead). If remote, we'd need HTTPS (~40–50 KB RAM per connection).
-- **Left-to-right color gradient** — the white-hot → green → black fade. Easy to replicate in RGB565 on the TFT. Keep?
+- **Server discovery:** Hardcoded IP in `secrets.h`, mDNS (`mainline.local`), or a DNS name?
-- **Glitch / noise effects** — the ░▒▓█ and katakana rain. Keep?
+- **WiFi credentials:** Use the existing multi-network setup from the doorbell firmware, or a specific SSID?
-- **Mic-reactive glitch intensity** — none of these boards have a microphone. Drop entirely, or substitute with something else (e.g. touch-reactive, or time-of-day reactive)?
-- **Auto-translation** — mainline translates headlines for region-specific sources via Google Translate. This requires HTTPS calls that are expensive on ESP32 (~40–50 KB RAM per TLS connection). Keep, pre-translate on a server, or drop?
-- **Poetry mode** — fetches full Gutenberg texts. These are large (100+ KB each). Cache to SD card, trim to a small set, or drop?
-- **Content filtering** — the sports/vapid regex filter. Trivial to keep.
-- **Boot sequence animation** — the typewriter-style boot log. Keep?
 
-### 4. Network environment
+### 4. ESP32 client repo
 
-- Will the board be on a WiFi network that can reach the public internet (RSS feeds, Google Translate, ntfy.sh)?
+The ESP32 sketch reuses `NetManager`, `IDisplayDriver`, and vendored display libraries from klubhaus-doorbell. Two options:
-- Is there a preferred SSID / network, or should it use the existing multi-network setup from the doorbell firmware?
 
-### 5. SD card availability
+- **klubhaus-doorbell repo** — natural fit as a new board target (`boards/esp32-mainline/`). Requires push access or a PR.
+- **mainline repo** — under `firmware/esp32-mainline/` with a vendored copy of KlubhausCore. Self-contained but duplicates shared code.
 
-All three boards have TF card slots but the doorbell firmware doesn't use them. A microSD card would be useful for caching fonts, pre-rendered glyph atlases, or translated headline buffers. **Is there an SD card in the board you'd want to target?**
+**Which repo should host the ESP32 client?**
+
+### 5. Display features
+
+With the renderer handling RSS fetching, translation, content filtering, and font rendering server-side, the remaining feature questions are about what the ESP32 displays locally:
+
+- **Left-to-right color gradient** — the white-hot → green → black fade, applied per-pixel on the ESP32. Keep?
+- **Glitch / noise effects** — random block characters and katakana rain between headlines. Keep?
+- **Glitch reactivity** — mainline.py uses a microphone (none on these boards). Substitute with touch-reactive, time-of-day reactive, or just random?
+- **Firehose mode** — dense data ticker at the bottom (only viable on Board 3). Want it?
+- **Boot sequence animation** — typewriter-style status log during startup. Keep?
+- **Poetry mode** — the server can serve Gutenberg text instead of RSS. Want both modes available, or just news?
+
+### 6. ntfy message queue
+
+The ESP32 polls `ntfy.sh/klubhaus_terminal_mainline` directly for messages that interrupt the news scroll.
+
+- **Is `klubhaus_terminal_mainline` the right topic name?** Or match the doorbell convention (`ALERT_klubhaus_topic`, etc.)?
+- **Who sends messages?** Just you (manual `curl`), a bot, other people?
+- **Display duration:** How long before auto-dismiss? The doorbell uses 120s for alerts. 30s for terminal messages? Touch-to-dismiss regardless?
+- **Priority levels?** ntfy supports 1–5. Should high-priority messages turn on the backlight if the display is OFF, or show in a different color?
+- **Message history on boot?** Show recent messages from the topic, or only messages arriving while running? The doorbell uses `?since=20s`. You might want `?since=5m` for a message board feel.
+
+### 7. Server-offline fallback
+
+If the renderer server goes down, what should the ESP32 do?
+
+- **A.** Show last cached headlines indefinitely (ntfy messages still work independently).
+- **B.** Show a "no signal" screen after a timeout, keep polling.
+- **C.** Fall back to ntfy messages + clock only.
+
+### 8. Scroll and layout
+
+- **Vertical scroll (like mainline.py)?** Or horizontal marquee?
+- **Speed:** mainline.py uses ~3.75s per headline. Same pace, or different for ambient display?
+- **Font height:** The server pre-renders at a configurable pixel height. On Board 3 (480px tall), 32–48px headlines would match mainline.py's feel. On Boards 1/2, 16–24px. What looks right to you?
+
+### 9. Font and aesthetics
+
+mainline.py uses `CSBishopDrawn-Italic.otf`. The server renders bitmaps with this font, so the ESP32 never needs the font file itself.
+
+- **Can you provide this font to whoever hosts the server?** Or should it fall back to a freely available alternative?
+- **Any license concern** with serving the rendered bitmaps (not the font file) over HTTP?
+
+### 10. SD card
+
+All three boards have TF card slots (unused by the doorbell firmware). An SD card would let the ESP32 cache headline bitmaps for instant boot and offline operation. **Is there an SD card in the board you'd pick?**
+
+---
+
+## Recent mainline.py changes and port implications
+
+*Updated after merge of latest `main` — three new features affect the port calculus.*
+
+### Local headline cache (`--refresh`)
+
+mainline.py now caches fetched headlines to a JSON file (`.mainline_cache_{MODE}.json`) and skips network fetches on subsequent runs unless `--refresh` is passed. This is good news for an ESP32 port:
+
+- **Faster boot.** The device could load cached headlines from flash/SD on power-up and start scrolling immediately, then refresh feeds in the background.
+- **Offline operation.** If WiFi drops, the device can still display cached content.
+- **Storage question.** A typical cache is ~200–500 headlines × ~100 bytes ≈ 20–50 KB of JSON. That fits comfortably in Board 3's 9 MB FAT partition or on SD. On Boards 1 & 2 (4 MB flash), it would need SPIFFS/LittleFS and would compete with app code for flash space.
+
+**Recommendation:** If targeting Boards 1 or 2, confirm how much flash is left after the app partition. Board 3's `app3M_fat9M_16MB` partition scheme already has 9 MB of FAT storage — more than enough.
+
+### Firehose mode (`--firehose`)
+
+mainline.py now has a `--firehose` flag that adds a 12-row dense data zone at the bottom of the viewport, cycling rapidly (every frame) through raw headline text, glitch noise, status lines, and headline fragments. This is the most demanding new feature for an ESP32 port:
+
+- **Frame rate.** The firehose zone redraws completely every frame at 20 FPS. On SPI displays (Boards 1 & 2), rewriting 12 rows × 320 or 480 pixels at 40 MHz SPI would consume a significant fraction of each 50 ms frame budget. On Board 3's RGB framebuffer, this is trivial.
+- **Randomness.** Each firehose line calls `random.choice()` over the full item pool. On ESP32, `esp_random()` is hardware-backed and fast, but iterating the full pool each frame needs the pool in RAM.
+- **Visual density.** At 320px wide (Boards 1 & 2), 12 rows of dense data may be illegible. At 800px (Board 3), it works.
+
+**Recommendation:** Firehose mode is only practical on Board 3. On Boards 1 & 2, consider either dropping it or replacing it with a single-line ticker.
+
+### Fixed 20 FPS frame timing
+
+The scroll loop now uses `time.monotonic()` with a 50 ms frame budget (`_FRAME_DT = 0.05`) and a scroll accumulator instead of sleeping per scroll step. This is actually a better fit for ESP32 than the old approach — it maps cleanly to a `millis()`-based main loop:
+
+```
+// ESP32 equivalent pattern
+void loop() {
+    uint32_t t0 = millis();
+    // ... render frame ...
+    uint32_t elapsed = millis() - t0;
+    if (elapsed < FRAME_DT_MS) delay(FRAME_DT_MS - elapsed);
+}
+```
+
+**Estimated per-frame budgets:**
+
+| | Board 1 (320×240) | Board 2 (320×480) | Board 3 (800×480) |
+|---|---|---|---|
+| Full-screen SPI flush | ~30 ms | ~60 ms | n/a (framebuffer) |
+| Partial update (12 rows) | ~4 ms | ~4 ms | n/a |
+| Remaining CPU budget (of 50 ms) | ~20 ms | ~0 ms (tight) | ~45 ms |
+
+Board 2 at 20 FPS with a full-screen redraw each frame would have essentially zero headroom. Partial updates (dirty-rect tracking) would be mandatory.
+
+### Port-relevant implications
+
+- **Firehose mode** is only practical on Board 3 (see §5 in Questions). On Boards 1 & 2, consider a single-line ticker instead.
+- **Headline caching** maps to the ESP32 storing bitmap data from the server to flash/SD for instant boot and offline fallback (see §10 in Questions).
+- **20 FPS frame timing** maps cleanly to a `millis()`-based ESP32 main loop. Board 2 would have zero headroom without partial updates.

mainline.py

@@ -39,7 +39,7 @@ sys.path.insert(0, str(next((_VENV / "lib").glob("python*/site-packages"))))
 
 import feedparser  # noqa: E402
 from PIL import Image, ImageDraw, ImageFont  # noqa: E402
-import random, time, re, signal, atexit, textwrap  # noqa: E402
+import random, time, re, signal, atexit, textwrap, threading  # noqa: E402
 try:
     import sounddevice as _sd
     import numpy as _np
@@ -58,13 +58,25 @@ MIC_THRESHOLD_DB = 50   # dB above which glitches intensify
 MODE = 'poetry' if '--poetry' in sys.argv or '-p' in sys.argv else 'news'
 FIREHOSE = '--firehose' in sys.argv
 
+# ntfy message queue
+NTFY_TOPIC = "https://ntfy.sh/klubhaus_terminal_mainline/json?since=20s&poll=1"
+NTFY_POLL_INTERVAL = 15   # seconds between polls
+MESSAGE_DISPLAY_SECS = 30 # how long a message holds the screen
+
 # Poetry/literature sources — public domain via Project Gutenberg
 POETRY_SOURCES = {
     "Whitman":    "https://www.gutenberg.org/cache/epub/1322/pg1322.txt",
     "Dickinson":  "https://www.gutenberg.org/cache/epub/12242/pg12242.txt",
-    "Thoreau":    "https://www.gutenberg.org/cache/epub/205/pg205.txt",
-    "Emerson":    "https://www.gutenberg.org/cache/epub/2944/pg2944.txt",
     "Whitman II": "https://www.gutenberg.org/cache/epub/8388/pg8388.txt",
+    "Rilke":      "https://www.gutenberg.org/cache/epub/38594/pg38594.txt",
+    "Pound":      "https://www.gutenberg.org/cache/epub/41162/pg41162.txt",
+    "Pound II":   "https://www.gutenberg.org/cache/epub/51992/pg51992.txt",
+    "Eliot":      "https://www.gutenberg.org/cache/epub/1567/pg1567.txt",
+    "Yeats":      "https://www.gutenberg.org/cache/epub/38877/pg38877.txt",
+    "Masters":    "https://www.gutenberg.org/cache/epub/1280/pg1280.txt",
+    "Baudelaire": "https://www.gutenberg.org/cache/epub/36098/pg36098.txt",
+    "Crane":      "https://www.gutenberg.org/cache/epub/40786/pg40786.txt",
+    "Poe":        "https://www.gutenberg.org/cache/epub/10031/pg10031.txt",
 }
 
 # ─── ANSI ─────────────────────────────────────────────────
@@ -516,6 +528,42 @@ def _start_mic():
         return False
 
 
+# ─── NTFY MESSAGE QUEUE ───────────────────────────────────
+_ntfy_message = None   # (title, body, monotonic_timestamp) or None
+_ntfy_lock = threading.Lock()
+
+
+def _start_ntfy_poller():
+    """Start background thread polling ntfy for messages."""
+    def _poll():
+        global _ntfy_message
+        while True:
+            try:
+                req = urllib.request.Request(
+                    NTFY_TOPIC, headers={"User-Agent": "mainline/0.1"})
+                resp = urllib.request.urlopen(req, timeout=10)
+                for line in resp.read().decode('utf-8', errors='replace').strip().split('\n'):
+                    if not line.strip():
+                        continue
+                    try:
+                        data = json.loads(line)
+                    except json.JSONDecodeError:
+                        continue
+                    if data.get("event") == "message":
+                        with _ntfy_lock:
+                            _ntfy_message = (
+                                data.get("title", ""),
+                                data.get("message", ""),
+                                time.monotonic(),
+                            )
+            except Exception:
+                pass
+            time.sleep(NTFY_POLL_INTERVAL)
+    t = threading.Thread(target=_poll, daemon=True)
+    t.start()
+    return True
+
+
 def _render_line(text, font=None):
     """Render a line of text as terminal rows using OTF font + half-blocks."""
     if font is None:
@@ -754,6 +802,7 @@ def _firehose_line(items, w):
 
 
 def stream(items):
+    global _ntfy_message
     random.shuffle(items)
     pool = list(items)
     seen = set()
@@ -781,12 +830,85 @@ def stream(items):
             noise_cache[cy] = noise(w) if random.random() < 0.15 else None
         return noise_cache[cy]
 
+    # Message color: bright cyan/white — distinct from headline greens
+    MSG_COLOR = "\033[1;38;5;87m"    # sky cyan
+    MSG_META  = "\033[38;5;245m"     # cool grey
+    MSG_BORDER = "\033[2;38;5;37m"   # dim teal
+    _msg_cache = (None, None)        # (cache_key, rendered_rows)
+
     while queued < HEADLINE_LIMIT or active:
         t0 = time.monotonic()
         w, h = tw(), th()
         fh = FIREHOSE_H if FIREHOSE else 0
         sh = h - fh
 
+        # ── Check for ntfy message ────────────────────────
+        msg_active = False
+        with _ntfy_lock:
+            if _ntfy_message is not None:
+                m_title, m_body, m_ts = _ntfy_message
+                if time.monotonic() - m_ts < MESSAGE_DISPLAY_SECS:
+                    msg_active = True
+                else:
+                    _ntfy_message = None  # expired
+
+        if msg_active:
+            # ── MESSAGE state: freeze scroll, render message ──
+            buf = []
+            # Render message text with OTF font (cached across frames)
+            display_text = m_body or m_title or "(empty)"
+            display_text = re.sub(r"\s+", " ", display_text.upper())
+            cache_key = (display_text, w)
+            if _msg_cache[0] != cache_key:
+                msg_rows = _big_wrap(display_text, w - 4)
+                msg_rows = _lr_gradient(msg_rows)
+                _msg_cache = (cache_key, msg_rows)
+            else:
+                msg_rows = _msg_cache[1]
+            # Center vertically in scroll zone
+            total_h = len(msg_rows) + 4  # +4 for border + meta + padding
+            y_off = max(0, (sh - total_h) // 2)
+            for r in range(sh):
+                ri = r - y_off
+                if ri == 0 or ri == total_h - 1:
+                    # Border lines
+                    bar = "─" * (w - 4)
+                    buf.append(f"\033[{r+1};1H  {MSG_BORDER}{bar}{RST}\033[K")
+                elif 1 <= ri <= len(msg_rows):
+                    ln = _vis_trunc(msg_rows[ri - 1], w)
+                    buf.append(f"\033[{r+1};1H  {ln}{RST}\033[K")
+                elif ri == len(msg_rows) + 1:
+                    # Title line (if present and different from body)
+                    if m_title and m_title != m_body:
+                        meta = f"  {MSG_META}\u2591 {m_title}{RST}"
+                    else:
+                        meta = ""
+                    buf.append(f"\033[{r+1};1H{meta}\033[K")
+                elif ri == len(msg_rows) + 2:
+                    # Source + timestamp
+                    elapsed_s = int(time.monotonic() - m_ts)
+                    remaining = max(0, MESSAGE_DISPLAY_SECS - elapsed_s)
+                    ts_str = datetime.now().strftime("%H:%M:%S")
+                    meta = f"  {MSG_META}\u2591 ntfy \u00b7 {ts_str} \u00b7 {remaining}s{RST}"
+                    buf.append(f"\033[{r+1};1H{meta}\033[K")
+                else:
+                    # Sparse noise outside the message
+                    if random.random() < 0.06:
+                        buf.append(f"\033[{r+1};1H{noise(w)}")
+                    else:
+                        buf.append(f"\033[{r+1};1H\033[K")
+            # Firehose keeps running during messages
+            if FIREHOSE and fh > 0:
+                for fr in range(fh):
+                    fline = _firehose_line(items, w)
+                    buf.append(f"\033[{sh + fr + 1};1H{fline}\033[K")
+            sys.stdout.buffer.write("".join(buf).encode())
+            sys.stdout.flush()
+            elapsed = time.monotonic() - t0
+            time.sleep(max(0, _FRAME_DT - elapsed))
+            continue
+
+        # ── SCROLL state: normal headline rendering ───────
         # Advance scroll on schedule
         scroll_accum += _FRAME_DT
         while scroll_accum >= scroll_interval:
@@ -937,6 +1059,8 @@ def main():
     mic_ok = _start_mic()
     if _HAS_MIC:
         boot_ln("Microphone", "ACTIVE" if mic_ok else "OFFLINE · check System Settings → Privacy → Microphone", mic_ok)
+    ntfy_ok = _start_ntfy_poller()
+    boot_ln("ntfy", "LISTENING" if ntfy_ok else "OFFLINE", ntfy_ok)
     if FIREHOSE:
         boot_ln("Firehose", "ENGAGED", True)