refactor: Create engine package, extracting data sources to sources.py, and add refactoring documentation.

2026-03-14 20:56:24 -07:00
parent 6a5a73fd88
commit dcc3718012
4 changed files with 325 additions and 0 deletions
--- a/mainline.md
+++ b/mainline.md
@@ -0,0 +1,178 @@
+# Refactor mainline\.py into modular package
+## Problem
+`mainline.py` is a single 1085\-line file with ~10 interleaved concerns\. This prevents:
+* Reusing the ntfy doorbell interrupt in other visualizers
+* Importing the render pipeline from `serve.py` \(future ESP32 HTTP server\)
+* Testing any concern in isolation
+* Porting individual layers to Rust independently
+## Target structure
+```warp-runnable-command
+mainline.py              # thin entrypoint: venv bootstrap → engine.app.main()
+engine/
+  __init__.py
+  config.py              # constants, CLI flags, glyph tables
+  sources.py             # FEEDS, POETRY_SOURCES, SOURCE_LANGS, _LOCATION_LANGS
+  terminal.py            # ANSI codes, tw/th, type_out, slow_print, boot_ln
+  filter.py              # HTML stripping, content filter (_SKIP_RE)
+  translate.py           # Google Translate wrapper + location→language detection
+  render.py              # OTF font loading, _render_line, _big_wrap, _lr_gradient, _make_block
+  effects.py             # noise, glitch_bar, _fade_line, _vis_trunc, _firehose_line, _next_headline
+  fetch.py               # RSS/Gutenberg fetching, cache load/save
+  ntfy.py                # NtfyPoller class — standalone, zero internal deps
+  mic.py                 # MicMonitor class — standalone
+  scroll.py              # stream() frame loop + message rendering
+  app.py                 # main(), TITLE art, boot sequence, signal handler
+```
+The package is named `engine/` to avoid a naming conflict with the `mainline.py` entrypoint\.
+## Module dependency graph
+```warp-runnable-command
+config      ← (nothing)
+sources     ← (nothing)
+terminal    ← (nothing)
+filter      ← (nothing)
+translate   ← sources
+render      ← config, terminal, sources
+effects     ← config, terminal, sources
+fetch       ← config, sources, filter, terminal
+ntfy        ← (nothing — stdlib only, fully standalone)
+mic         ← (nothing — sounddevice only)
+scroll      ← config, terminal, render, effects, ntfy, mic
+app         ← everything above
+```
+Critical property: **ntfy\.py and mic\.py have zero internal dependencies**, making ntfy reusable by any visualizer\.
+## Module details
+### mainline\.py \(entrypoint — slimmed down\)
+Keeps only the venv bootstrap \(lines 10\-38\) which must run before any third\-party imports\. After bootstrap, delegates to `engine.app.main()`\.
+### engine/config\.py
+From current mainline\.py:
+* `HEADLINE_LIMIT`, `FEED_TIMEOUT`, `MIC_THRESHOLD_DB` \(lines 55\-57\)
+* `MODE`, `FIREHOSE` CLI flag parsing \(lines 58\-59\)
+* `NTFY_TOPIC`, `NTFY_POLL_INTERVAL`, `MESSAGE_DISPLAY_SECS` \(lines 62\-64\)
+* `_FONT_PATH`, `_FONT_SZ`, `_RENDER_H` \(lines 147\-150\)
+* `_SCROLL_DUR`, `_FRAME_DT`, `FIREHOSE_H` \(lines 505\-507\)
+* `GLITCH`, `KATA` glyph tables \(lines 143\-144\)
+### engine/sources\.py
+Pure data, no logic:
+* `FEEDS` dict \(lines 102\-140\)
+* `POETRY_SOURCES` dict \(lines 67\-80\)
+* `SOURCE_LANGS` dict \(lines 258\-266\)
+* `_LOCATION_LANGS` dict \(lines 269\-289\)
+* `_SCRIPT_FONTS` dict \(lines 153\-165\)
+* `_NO_UPPER` set \(line 167\)
+### engine/terminal\.py
+ANSI primitives and terminal I/O:
+* All ANSI constants: `RST`, `BOLD`, `DIM`, `G_HI`, `G_MID`, `G_LO`, `G_DIM`, `W_COOL`, `W_DIM`, `W_GHOST`, `C_DIM`, `CLR`, `CURSOR_OFF`, `CURSOR_ON` \(lines 83\-99\)
+* `tw()`, `th()` \(lines 223\-234\)
+* `type_out()`, `slow_print()`, `boot_ln()` \(lines 355\-386\)
+### engine/filter\.py
+* `_Strip` HTML parser class \(lines 205\-214\)
+* `strip_tags()` \(lines 217\-220\)
+* `_SKIP_RE` compiled regex \(lines 322\-346\)
+* `_skip()` predicate \(lines 349\-351\)
+### engine/translate\.py
+* `_TRANSLATE_CACHE` \(line 291\)
+* `_detect_location_language()` \(lines 294\-300\) — imports `_LOCATION_LANGS` from sources
+* `_translate_headline()` \(lines 303\-319\)
+### engine/render\.py
+The OTF→terminal pipeline\. This is exactly what `serve.py` will import to produce 1\-bit bitmaps for the ESP32\.
+* `_GRAD_COLS` gradient table \(lines 169\-182\)
+* `_font()`, `_font_for_lang()` with lazy\-load \+ cache \(lines 185\-202\)
+* `_render_line()` — OTF text → half\-block terminal rows \(lines 567\-605\)
+* `_big_wrap()` — word\-wrap \+ render \(lines 608\-636\)
+* `_lr_gradient()` — apply left→right color gradient \(lines 639\-656\)
+* `_make_block()` — composite: translate → render → colorize a headline \(lines 718\-756\)\. Imports from translate, sources\.
+### engine/effects\.py
+Visual effects applied during the frame loop:
+* `noise()` \(lines 237\-245\)
+* `glitch_bar()` \(lines 248\-252\)
+* `_fade_line()` — probabilistic character dissolve \(lines 659\-680\)
+* `_vis_trunc()` — ANSI\-aware width truncation \(lines 683\-701\)
+* `_firehose_line()` \(lines 759\-801\) — imports config\.MODE, sources\.FEEDS/POETRY\_SOURCES
+* `_next_headline()` — pool management \(lines 704\-715\)
+### engine/fetch\.py
+* `fetch_feed()` \(lines 390\-396\)
+* `fetch_all()` \(lines 399\-426\) — imports filter\.\_skip, filter\.strip\_tags, terminal\.boot\_ln
+* `_fetch_gutenberg()` \(lines 429\-456\)
+* `fetch_poetry()` \(lines 459\-472\)
+* `_cache_path()`, `_load_cache()`, `_save_cache()` \(lines 476\-501\)
+### engine/ntfy\.py — standalone, reusable
+Refactored from the current globals \+ thread \(lines 531\-564\) and the message rendering section of `stream()` \(lines 845\-909\) into a class:
+```python
+class NtfyPoller:
+    def __init__(self, topic_url, poll_interval=15, display_secs=30):
+        ...
+    def start(self):
+        """Start background polling thread."""
+    def get_active_message(self):
+        """Return (title, body, timestamp) if a message is active and not expired, else None."""
+    def dismiss(self):
+        """Manually dismiss current message."""
+```
+Dependencies: `urllib.request`, `json`, `threading`, `time` — all stdlib\. No internal imports\.
+Other visualizers use it like:
+```python
+from engine.ntfy import NtfyPoller
+poller = NtfyPoller("https://ntfy.sh/my_topic/json?since=20s&poll=1")
+poller.start()
+# in render loop:
+msg = poller.get_active_message()
+if msg:
+    title, body, ts = msg
+    render_my_message(title, body)  # visualizer-specific
+```
+### engine/mic\.py — standalone
+Refactored from the current globals \(lines 508\-528\) into a class:
+```python
+class MicMonitor:
+    def __init__(self, threshold_db=50):
+        ...
+    def start(self) -> bool:
+        """Start background mic stream. Returns False if unavailable."""
+    def stop(self):
+        ...
+    @property
+    def db(self) -> float:
+        """Current RMS dB level."""
+    @property
+    def excess(self) -> float:
+        """dB above threshold (clamped to 0)."""
+```
+Dependencies: `sounddevice`, `numpy` \(both optional — graceful fallback\)\.
+### engine/scroll\.py
+The `stream()` function \(lines 804\-990\)\. Receives its dependencies via arguments or imports:
+* `stream(items, ntfy_poller, mic_monitor, config)` or similar
+* Message rendering \(lines 855\-909\) stays here since it's terminal\-display\-specific — a different visualizer would render messages differently
+### engine/app\.py
+The orchestrator:
+* `TITLE` ASCII art \(lines 994\-1001\)
+* `main()` \(lines 1004\-1084\): CLI handling, signal setup, boot animation, fetch, wire up ntfy/mic/scroll
+## Execution order
+### Step 1: Create engine/ package skeleton
+Create `engine/__init__.py` and all empty module files\.
+### Step 2: Extract pure data modules \(zero\-dep\)
+Move constants and data dicts into `config.py`, `sources.py`\. These have no logic dependencies\.
+### Step 3: Extract terminal\.py
+Move ANSI codes and terminal I/O helpers\. No internal deps\.
+### Step 4: Extract filter\.py and translate\.py
+Both are small, self\-contained\. translate imports from sources\.
+### Step 5: Extract render\.py
+Font loading \+ the OTF→half\-block pipeline\. Imports from config, terminal, sources\. This is the module `serve.py` will later import\.
+### Step 6: Extract effects\.py
+Visual effects\. Imports from config, terminal, sources\.
+### Step 7: Extract fetch\.py
+Feed/Gutenberg fetching \+ caching\. Imports from config, sources, filter, terminal\.
+### Step 8: Extract ntfy\.py and mic\.py
+Refactor globals\+threads into classes\. Zero internal deps\.
+### Step 9: Extract scroll\.py
+The frame loop\. Last to extract because it depends on everything above\.
+### Step 10: Extract app\.py
+The `main()` function, boot sequence, signal handler\. Wire up all modules\.
+### Step 11: Slim down mainline\.py
+Keep only venv bootstrap \+ `from engine.app import main; main()`\.
+### Step 12: Verify
+Run `python3 mainline.py`, `python3 mainline.py --poetry`, and `python3 mainline.py --firehose` to confirm identical behavior\. No behavioral changes in this refactor\.
+## What this enables
+* **serve\.py** \(future\): `from engine.render import _render_line, _big_wrap` \+ `from engine.fetch import fetch_all` — imports the pipeline directly
+* **Other visualizers**: `from engine.ntfy import NtfyPoller` — doorbell feature with no coupling to mainline's scroll engine
+* **Rust port**: Clear boundaries for what to port first \(ntfy client, render pipeline\) vs what stays in Python \(fetching, caching — the server side\)
+* **Testing**: Each module can be unit\-tested in isolation