sideline/docs/Refactor mainline.md

Refactor mainline.py into modular package

Problem

mainline.py is a single 1085-line file with ~10 interleaved concerns. This prevents:

• Reusing the ntfy doorbell interrupt in other visualizers
• Importing the render pipeline from serve.py (future ESP32 HTTP server)
• Testing any concern in isolation
• Porting individual layers to Rust independently

Target structure

mainline.py              # thin entrypoint: venv bootstrap → engine.app.main()
engine/
  __init__.py
  config.py              # constants, CLI flags, glyph tables
  sources.py             # FEEDS, POETRY_SOURCES, SOURCE_LANGS, _LOCATION_LANGS
  terminal.py            # ANSI codes, tw/th, type_out, slow_print, boot_ln
  filter.py              # HTML stripping, content filter (_SKIP_RE)
  translate.py           # Google Translate wrapper + location→language detection
  render.py              # OTF font loading, _render_line, _big_wrap, _lr_gradient, _make_block
  effects.py             # noise, glitch_bar, _fade_line, _vis_trunc, _firehose_line, _next_headline
  fetch.py               # RSS/Gutenberg fetching, cache load/save
  ntfy.py                # NtfyPoller class — standalone, zero internal deps
  mic.py                 # MicMonitor class — standalone
  scroll.py              # stream() frame loop + message rendering
  app.py                 # main(), TITLE art, boot sequence, signal handler

The package is named engine/ to avoid a naming conflict with the mainline.py entrypoint.

Module dependency graph

config      ← (nothing)
sources     ← (nothing)
terminal    ← (nothing)
filter      ← (nothing)
translate   ← sources
render      ← config, terminal, sources
effects     ← config, terminal, sources
fetch       ← config, sources, filter, terminal
ntfy        ← (nothing — stdlib only, fully standalone)
mic         ← (nothing — sounddevice only)
scroll      ← config, terminal, render, effects, ntfy, mic
app         ← everything above

Critical property: ntfy.py and mic.py have zero internal dependencies, making ntfy reusable by any visualizer.

Module details

mainline.py (entrypoint — slimmed down)

Keeps only the venv bootstrap (lines 10-38) which must run before any third-party imports. After bootstrap, delegates to engine.app.main().

engine/config.py

From current mainline.py:

• HEADLINE_LIMIT, FEED_TIMEOUT, MIC_THRESHOLD_DB (lines 55-57)
• MODE, FIREHOSE CLI flag parsing (lines 58-59)
• NTFY_TOPIC, NTFY_POLL_INTERVAL, MESSAGE_DISPLAY_SECS (lines 62-64)
• _FONT_PATH, _FONT_SZ, _RENDER_H (lines 147-150)
• _SCROLL_DUR, _FRAME_DT, FIREHOSE_H (lines 505-507)
• GLITCH, KATA glyph tables (lines 143-144)

engine/sources.py

Pure data, no logic:

• FEEDS dict (lines 102-140)
• POETRY_SOURCES dict (lines 67-80)
• SOURCE_LANGS dict (lines 258-266)
• _LOCATION_LANGS dict (lines 269-289)
• _SCRIPT_FONTS dict (lines 153-165)
• _NO_UPPER set (line 167)

engine/terminal.py

ANSI primitives and terminal I/O:

• All ANSI constants: RST, BOLD, DIM, G_HI, G_MID, G_LO, G_DIM, W_COOL, W_DIM, W_GHOST, C_DIM, CLR, CURSOR_OFF, CURSOR_ON (lines 83-99)
• tw(), th() (lines 223-234)
• type_out(), slow_print(), boot_ln() (lines 355-386)

engine/filter.py

• _Strip HTML parser class (lines 205-214)
• strip_tags() (lines 217-220)
• _SKIP_RE compiled regex (lines 322-346)
• _skip() predicate (lines 349-351)

engine/translate.py

• _TRANSLATE_CACHE (line 291)
• _detect_location_language() (lines 294-300) — imports _LOCATION_LANGS from sources
• _translate_headline() (lines 303-319)

engine/render.py

The OTF→terminal pipeline. This is exactly what serve.py will import to produce 1-bit bitmaps for the ESP32.

• _GRAD_COLS gradient table (lines 169-182)
• _font(), _font_for_lang() with lazy-load + cache (lines 185-202)
• _render_line() — OTF text → half-block terminal rows (lines 567-605)
• _big_wrap() — word-wrap + render (lines 608-636)
• _lr_gradient() — apply left→right color gradient (lines 639-656)
• _make_block() — composite: translate → render → colorize a headline (lines 718-756). Imports from translate, sources.

engine/effects.py

Visual effects applied during the frame loop:

• noise() (lines 237-245)
• glitch_bar() (lines 248-252)
• _fade_line() — probabilistic character dissolve (lines 659-680)
• _vis_trunc() — ANSI-aware width truncation (lines 683-701)
• _firehose_line() (lines 759-801) — imports config.MODE, sources.FEEDS/POETRY_SOURCES
• _next_headline() — pool management (lines 704-715)

engine/fetch.py

• fetch_feed() (lines 390-396)
• fetch_all() (lines 399-426) — imports filter._skip, filter.strip_tags, terminal.boot_ln
• _fetch_gutenberg() (lines 429-456)
• fetch_poetry() (lines 459-472)
• _cache_path(), _load_cache(), _save_cache() (lines 476-501)

engine/ntfy.py — standalone, reusable

Refactored from the current globals + thread (lines 531-564) and the message rendering section of stream() (lines 845-909) into a class:

class NtfyPoller:
    def __init__(self, topic_url, poll_interval=15, display_secs=30):
        ...
    def start(self):
        """Start background polling thread."""
    def get_active_message(self):
        """Return (title, body, timestamp) if a message is active and not expired, else None."""
    def dismiss(self):
        """Manually dismiss current message."""

Dependencies: urllib.request, json, threading, time — all stdlib. No internal imports. Other visualizers use it like:

from engine.ntfy import NtfyPoller
poller = NtfyPoller("https://ntfy.sh/my_topic/json?since=20s&poll=1")
poller.start()
# in render loop:
msg = poller.get_active_message()
if msg:
    title, body, ts = msg
    render_my_message(title, body)  # visualizer-specific

engine/mic.py — standalone

Refactored from the current globals (lines 508-528) into a class:

class MicMonitor:
    def __init__(self, threshold_db=50):
        ...
    def start(self) -> bool:
        """Start background mic stream. Returns False if unavailable."""
    def stop(self):
        ...
    @property
    def db(self) -> float:
        """Current RMS dB level."""
    @property
    def excess(self) -> float:
        """dB above threshold (clamped to 0)."""

Dependencies: sounddevice, numpy (both optional — graceful fallback).

engine/scroll.py

The stream() function (lines 804-990). Receives its dependencies via arguments or imports:

• stream(items, ntfy_poller, mic_monitor, config) or similar
• Message rendering (lines 855-909) stays here since it's terminal-display-specific — a different visualizer would render messages differently

engine/app.py

The orchestrator:

• TITLE ASCII art (lines 994-1001)
• main() (lines 1004-1084): CLI handling, signal setup, boot animation, fetch, wire up ntfy/mic/scroll

Execution order

Step 1: Create engine/ package skeleton

Create engine/__init__.py and all empty module files.

Step 2: Extract pure data modules (zero-dep)

Move constants and data dicts into config.py, sources.py. These have no logic dependencies.

Step 3: Extract terminal.py

Move ANSI codes and terminal I/O helpers. No internal deps.

Step 4: Extract filter.py and translate.py

Both are small, self-contained. translate imports from sources.

Step 5: Extract render.py

Font loading + the OTF→half-block pipeline. Imports from config, terminal, sources. This is the module serve.py will later import.

Step 6: Extract effects.py

Visual effects. Imports from config, terminal, sources.

Step 7: Extract fetch.py

Feed/Gutenberg fetching + caching. Imports from config, sources, filter, terminal.

Step 8: Extract ntfy.py and mic.py

Refactor globals+threads into classes. Zero internal deps.

Step 9: Extract scroll.py

The frame loop. Last to extract because it depends on everything above.

Step 10: Extract app.py

The main() function, boot sequence, signal handler. Wire up all modules.

Step 11: Slim down mainline.py

Keep only venv bootstrap + from engine.app import main; main().

Step 12: Verify

Run python3 mainline.py, python3 mainline.py --poetry, and python3 mainline.py --firehose to confirm identical behavior. No behavioral changes in this refactor.

What this enables

• serve.py (future): from engine.render import _render_line, _big_wrap + from engine.fetch import fetch_all — imports the pipeline directly
• Other visualizers: from engine.ntfy import NtfyPoller — doorbell feature with no coupling to mainline's scroll engine
• Rust port: Clear boundaries for what to port first (ntfy client, render pipeline) vs what stays in Python (fetching, caching — the server side)
• Testing: Each module can be unit-tested in isolation