Merge pull request 'refactor to improve testability and modularization of the mainline terminal application' (#24) from david/Mainline:testability_modularization into main

Reviewed-on: #24

README.md

@@ -6,7 +6,9 @@ A full-screen terminal news ticker that renders live global headlines in large O
 
 ---
 
-## Run
+## Using
+
+### Run
 
 ```bash
 python3 mainline.py                      # news stream
@@ -20,11 +22,15 @@ python3 mainline.py --font-dir ~/fonts   # scan a different font folder
 python3 mainline.py --font-index 1       # select face index within a collection
 ```
 
-First run bootstraps a local `.mainline_venv/` and installs deps (`feedparser`, `Pillow`, `sounddevice`, `numpy`). Subsequent runs start immediately, loading from cache.
+Or with uv:
 
----
+```bash
+uv run mainline.py
+```
 
-## Config
+First run bootstraps a local `.mainline_venv/` and installs deps (`feedparser`, `Pillow`, `sounddevice`, `numpy`). Subsequent runs start immediately, loading from cache. With uv, run `uv sync` or `uv sync --all-extras` (includes mic support) instead.
+
+### Config
 
 All constants live in `engine/config.py`:
 
@@ -44,23 +50,41 @@ All constants live in `engine/config.py`:
 | `FRAME_DT` | `0.05` | Frame interval in seconds (20 FPS) |
 | `GRAD_SPEED` | `0.08` | Gradient sweep speed (cycles/sec, ~12s full sweep) |
 | `FIREHOSE_H` | `12` | Firehose zone height (terminal rows) |
-| `NTFY_TOPIC` | klubhaus URL | ntfy.sh JSON endpoint to poll |
-| `NTFY_POLL_INTERVAL` | `15` | Seconds between ntfy polls |
+| `NTFY_TOPIC` | klubhaus URL | ntfy.sh JSON stream endpoint |
+| `NTFY_RECONNECT_DELAY` | `5` | Seconds before reconnecting after a dropped SSE stream |
 | `MESSAGE_DISPLAY_SECS` | `30` | How long an ntfy message holds the screen |
 
----
+### Feeds
 
-## Fonts
+~25 sources across four categories: **Science & Technology**, **Economics & Business**, **World & Politics**, **Culture & Ideas**. Add or swap feeds in `engine/sources.py` → `FEEDS`.
 
-A `fonts/` directory is bundled with demo faces (AlphatronDemo, CSBishopDrawn, CyberformDemo, KATA, Microbots, Neoform, Pixel Sparta, Robocops, Xeonic, and others). On startup, an interactive picker lists all discovered faces with a live half-block preview rendered at your configured size.
+**Poetry mode** pulls from Project Gutenberg: Whitman, Dickinson, Thoreau, Emerson. Sources are in `engine/sources.py` → `POETRY_SOURCES`.
+
+### Fonts
+
+A `fonts/` directory is bundled with demo faces (AgorTechnoDemo, AlphatronDemo, CSBishopDrawn, CubaTechnologyDemo, CyberformDemo, KATA, Microbots, ModernSpaceDemo, Neoform, Pixel Sparta, RaceHugoDemo, Resond, Robocops, Synthetix, Xeonic, and others). On startup, an interactive picker lists all discovered faces with a live half-block preview rendered at your configured size.
 
 Navigation: `↑`/`↓` or `j`/`k` to move, `Enter` or `q` to select. The selected face persists for that session.
 
 To add your own fonts, drop `.otf`, `.ttf`, or `.ttc` files into `fonts/` (or point `--font-dir` at any other folder). Font collections (`.ttc`, multi-face `.otf`) are enumerated face-by-face.
 
+### ntfy.sh
+
+Mainline polls a configurable ntfy.sh topic in the background. When a message arrives, the scroll pauses and the message renders full-screen for `MESSAGE_DISPLAY_SECS` seconds, then the stream resumes.
+
+To push a message:
+
+```bash
+curl -d "Body text" -H "Title: Alert title" https://ntfy.sh/your_topic
+```
+
+Update `NTFY_TOPIC` in `engine/config.py` to point at your own topic.
+
 ---
 
-## How it works
+## Internals
+
+### How it works
 
 - On launch, the font picker scans `fonts/` and presents a live-rendered TUI for face selection; `--no-font-picker` skips directly to stream
 - Feeds are fetched and filtered on startup (sports and vapid content stripped); results are cached to `.mainline_cache_news.json` / `.mainline_cache_poetry.json` for fast restarts
@@ -69,11 +93,9 @@ To add your own fonts, drop `.otf`, `.ttf`, or `.ttc` files into `fonts/` (or po
 - Subject-region detection runs a regex pass on each headline; matches trigger a Google Translate call and font swap to the appropriate script (CJK, Arabic, Devanagari, etc.) using macOS system fonts
 - The mic stream runs in a background thread, feeding RMS dB into the glitch probability calculation each frame
 - The viewport scrolls through a virtual canvas of pre-rendered blocks; fade zones at top and bottom dissolve characters probabilistically
-- An ntfy.sh poller runs in a background thread; incoming messages interrupt the scroll and render full-screen until dismissed or expired
+- An ntfy.sh SSE stream runs in a background thread; incoming messages interrupt the scroll and render full-screen until dismissed or expired
 
----
-
-## Architecture
+### Architecture
 
 `mainline.py` is a thin entrypoint (venv bootstrap → `engine.app.main()`). All logic lives in the `engine/` package:
 
@@ -85,7 +107,7 @@ engine/
   sources.py        FEEDS, POETRY_SOURCES, language/script maps
   terminal.py       ANSI codes, tw/th, type_out, boot_ln
   filter.py         HTML stripping, content filter
-  translate.py       Google Translate wrapper + region detection
+  translate.py      Google Translate wrapper + region detection
   render.py         OTF → half-block pipeline (SSAA, gradient)
   effects.py        noise, glitch_bar, fade, firehose
   fetch.py          RSS/Gutenberg fetching + cache load/save
@@ -106,37 +128,108 @@ engine/
 
 ---
 
-## Feeds
+## Extending
 
-~25 sources across four categories: **Science & Technology**, **Economics & Business**, **World & Politics**, **Culture & Ideas**. Add or swap feeds in `engine/sources.py` → `FEEDS`.
+`ntfy.py` and `mic.py` are fully standalone and designed to be reused by any terminal visualizer. `engine.render` is the importable rendering pipeline for non-terminal targets.
 
-**Poetry mode** pulls from Project Gutenberg: Whitman, Dickinson, Thoreau, Emerson. Sources are in `engine/sources.py` → `POETRY_SOURCES`.
-
----
-
-## ntfy.sh Integration
-
-Mainline polls a configurable ntfy.sh topic in the background. When a message arrives, the scroll pauses and the message renders full-screen for `MESSAGE_DISPLAY_SECS` seconds, then the stream resumes.
-
-To push a message:
-
-```bash
-curl -d "Body text" -H "Title: Alert title" https://ntfy.sh/your_topic
-```
-
-Update `NTFY_TOPIC` in `engine/config.py` to point at your own topic. The `NtfyPoller` class is fully standalone and can be reused by other visualizers:
+### NtfyPoller
 
 ```python
 from engine.ntfy import NtfyPoller
-poller = NtfyPoller("https://ntfy.sh/my_topic/json?since=20s&poll=1")
+
+poller = NtfyPoller("https://ntfy.sh/my_topic/json")
 poller.start()
-# in render loop:
-msg = poller.get_active_message()  # returns (title, body, timestamp) or None
+
+# in your render loop:
+msg = poller.get_active_message()  # → (title, body, timestamp) or None
+if msg:
+    title, body, ts = msg
+    render_my_message(title, body)  # visualizer-specific
 ```
 
+Dependencies: `urllib.request`, `json`, `threading`, `time` — stdlib only. The `since=` parameter is managed automatically on reconnect.
+
+### MicMonitor
+
+```python
+from engine.mic import MicMonitor
+
+mic = MicMonitor(threshold_db=50)
+result = mic.start()   # None = sounddevice unavailable; False = stream failed; True = ok
+if result:
+    excess = mic.excess    # dB above threshold, clamped to 0
+    db = mic.db            # raw RMS dB level
+```
+
+Dependencies: `sounddevice`, `numpy` — both optional; degrades gracefully if unavailable.
+
+### Render pipeline
+
+`engine.render` exposes the OTF → raster pipeline independently of the terminal scroll loop. The planned `serve.py` extension will import it directly to pre-render headlines as 1-bit bitmaps for an ESP32 thin client:
+
+```python
+# planned — serve.py does not yet exist
+from engine.render import render_line, big_wrap
+from engine.fetch import fetch_all
+
+headlines = fetch_all()
+for h in headlines:
+    rows = big_wrap(h.text, font, width=800)  # list of half-block rows
+    # threshold to 1-bit, pack bytes, serve over HTTP
+```
+
+See `Mainline Renderer + ntfy Message Queue for ESP32.md` for the full server + thin client architecture.
+
 ---
 
-## Ideas / Future
+## Development
+
+### Setup
+
+Requires Python 3.10+ and [uv](https://docs.astral.sh/uv/).
+
+```bash
+uv sync                              # minimal (no mic)
+uv sync --all-extras                 # with mic support (sounddevice + numpy)
+uv sync --all-extras --group dev     # full dev environment
+```
+
+### Tasks
+
+With [mise](https://mise.jdx.dev/):
+
+```bash
+mise run test            # run test suite
+mise run test-cov        # run with coverage report
+mise run lint            # ruff check
+mise run lint-fix        # ruff check --fix
+mise run format          # ruff format
+mise run run             # uv run mainline.py
+mise run run-poetry      # uv run mainline.py --poetry
+mise run run-firehose    # uv run mainline.py --firehose
+```
+
+### Testing
+
+Tests live in `tests/` and cover `config`, `filter`, `mic`, `ntfy`, `sources`, and `terminal`.
+
+```bash
+uv run pytest
+uv run pytest --cov=engine --cov-report=term-missing
+```
+
+### Linting
+
+```bash
+uv run ruff check engine/ mainline.py
+uv run ruff format engine/ mainline.py
+```
+
+Pre-commit hooks run lint automatically via `hk`.
+
+---
+
+## Roadmap
 
 ### Performance
 - **Concurrent feed fetching** — startup currently blocks sequentially on ~25 HTTP requests; `concurrent.futures.ThreadPoolExecutor` would cut load time to the slowest single feed
@@ -163,5 +256,4 @@ msg = poller.get_active_message()  # returns (title, body, timestamp) or None
 
 ---
 
-*macOS only (script/system font paths for translation are hardcoded). Primary display font is user-selectable via the bundled `fonts/` picker. Python 3.9+.*
-# test
+*macOS only (script/system font paths for translation are hardcoded). Primary display font is user-selectable via the bundled `fonts/` picker. Python 3.10+.*

docs/superpowers/specs/2026-03-15-readme-update-design.md

@@ -0,0 +1,145 @@
+# README Update Design — 2026-03-15
+
+## Goal
+
+Restructure and expand `README.md` to:
+1. Align with the current codebase (Python 3.10+, uv/mise/pytest/ruff toolchain, 6 new fonts)
+2. Add extensibility-focused content (`Extending` section)
+3. Add developer workflow coverage (`Development` section)
+4. Improve navigability via top-level grouping (Approach C)
+
+---
+
+## Proposed Structure
+
+```
+# MAINLINE
+> tagline + description
+
+## Using
+  ### Run
+  ### Config
+  ### Feeds
+  ### Fonts
+  ### ntfy.sh
+
+## Internals
+  ### How it works
+  ### Architecture
+
+## Extending
+  ### NtfyPoller
+  ### MicMonitor
+  ### Render pipeline
+
+## Development
+  ### Setup
+  ### Tasks
+  ### Testing
+  ### Linting
+
+## Roadmap
+
+---
+*footer*
+```
+
+---
+
+## Section-by-section design
+
+### Using
+
+All existing content preserved verbatim. Two changes:
+- **Run**: add `uv run mainline.py` as an alternative invocation; expand bootstrap note to mention `uv sync` / `uv sync --all-extras`
+- **ntfy.sh**: remove `NtfyPoller` reuse code example (moves to Extending); keep push instructions and topic config
+
+Subsections moved into Using (currently standalone):
+- `Feeds` — it's configuration, not a concept
+- `ntfy.sh` (usage half)
+
+### Internals
+
+All existing content preserved verbatim. One change:
+- **Architecture**: append `tests/` directory listing to the module tree
+
+### Extending
+
+Entirely new section. Three subsections:
+
+**NtfyPoller**
+- Minimal working import + usage example
+- Note: stdlib only dependencies
+
+```python
+from engine.ntfy import NtfyPoller
+
+poller = NtfyPoller("https://ntfy.sh/my_topic/json?since=20s&poll=1")
+poller.start()
+
+# in your render loop:
+msg = poller.get_active_message()  # → (title, body, timestamp) or None
+if msg:
+    title, body, ts = msg
+    render_my_message(title, body)  # visualizer-specific
+```
+
+**MicMonitor**
+- Minimal working import + usage example
+- Note: sounddevice/numpy optional, degrades gracefully
+
+```python
+from engine.mic import MicMonitor
+
+mic = MicMonitor(threshold_db=50)
+if mic.start():                # returns False if sounddevice unavailable
+    excess = mic.excess        # dB above threshold, clamped to 0
+    db = mic.db                # raw RMS dB level
+```
+
+**Render pipeline**
+- Brief prose about `engine.render` as importable pipeline
+- Minimal sketch of serve.py / ESP32 usage pattern
+- Reference to `Mainline Renderer + ntfy Message Queue for ESP32.md`
+
+### Development
+
+Entirely new section. Four subsections:
+
+**Setup**
+- Hard requirements: Python 3.10+, uv
+- `uv sync` / `uv sync --all-extras` / `uv sync --group dev`
+
+**Tasks** (via mise)
+- `mise run test`, `test-cov`, `lint`, `lint-fix`, `format`, `run`, `run-poetry`, `run-firehose`
+
+**Testing**
+- Tests in `tests/` covering config, filter, mic, ntfy, sources, terminal
+- `uv run pytest` and `uv run pytest --cov=engine --cov-report=term-missing`
+
+**Linting**
+- `uv run ruff check` and `uv run ruff format`
+- Note: pre-commit hooks run lint via `hk`
+
+### Roadmap
+
+Existing `## Ideas / Future` content preserved verbatim. Only change: rename heading to `## Roadmap`.
+
+### Footer
+
+Update `Python 3.9+` → `Python 3.10+`.
+
+---
+
+## Files changed
+
+- `README.md` — restructured and expanded as above
+- No other files
+
+---
+
+## What is not changing
+
+- All existing prose, examples, and config table values — preserved verbatim where retained
+- The Ideas/Future content — kept intact under the new Roadmap heading
+- The cyberpunk voice and terse style of the existing README