David Gwilliam e0bbfea26c refactor: consolidate pipeline architecture with unified data source system
MAJOR REFACTORING: Consolidate duplicated pipeline code and standardize on
capability-based dependency resolution. This is a significant but backwards-compatible
restructuring that improves maintainability and extensibility.

## ARCHITECTURE CHANGES

### Data Sources Consolidation
- Move engine/sources_v2.py → engine/data_sources/sources.py
- Move engine/pipeline_sources/ → engine/data_sources/
- Create unified DataSource ABC with common interface:
  * fetch() - idempotent data retrieval
  * get_items() - cached access with automatic refresh
  * refresh() - force cache invalidation
  * is_dynamic - indicate streaming vs static sources
- Support for SourceItem dataclass (content, source, timestamp, metadata)

### Display Backend Improvements
- Update all 7 display backends to use new import paths
- Terminal: Improve dimension detection and handling
- WebSocket: Better error handling and client lifecycle
- Sixel: Refactor graphics rendering
- Pygame: Modernize event handling
- Kitty: Add protocol support for inline images
- Multi: Ensure proper forwarding to all backends
- Null: Maintain testing backend functionality

### Pipeline Adapter Consolidation
- Refactor adapter stages for clarity and flexibility
- RenderStage now handles both item-based and buffer-based rendering
- Add SourceItemsToBufferStage for converting data source items
- Improve DataSourceStage to work with all source types
- Add DisplayStage wrapper for display backends

### Camera & Viewport Refinements
- Update Camera class for new architecture
- Improve viewport dimension detection
- Better handling of resize events across backends

### New Effect Plugins
- border.py: Frame rendering effect with configurable style
- crop.py: Viewport clipping effect for selective display
- tint.py: Color filtering effect for atmosphere

### Tests & Quality
- Add test_border_effect.py with comprehensive border tests
- Add test_crop_effect.py with viewport clipping tests
- Add test_tint_effect.py with color filtering tests
- Update test_pipeline.py for new architecture
- Update test_pipeline_introspection.py for new data source location
- All 463 tests pass with 56% coverage
- Linting: All checks pass with ruff

### Removals (Code Cleanup)
- Delete engine/benchmark.py (deprecated performance testing)
- Delete engine/pipeline_sources/__init__.py (moved to data_sources)
- Delete engine/sources_v2.py (replaced by data_sources/sources.py)
- Update AGENTS.md to reflect new structure

### Import Path Updates
- Update engine/pipeline/controller.py::create_default_pipeline()
  * Old: from engine.sources_v2 import HeadlinesDataSource
  * New: from engine.data_sources.sources import HeadlinesDataSource
- All display backends import from new locations
- All tests import from new locations

## BACKWARDS COMPATIBILITY

This refactoring is intended to be backwards compatible:
- Pipeline execution unchanged (DAG-based with capability matching)
- Effect plugins unchanged (EffectPlugin interface same)
- Display protocol unchanged (Display duck-typing works as before)
- Config system unchanged (presets.toml format same)

## TESTING

- 463 tests pass (0 failures, 19 skipped)
- Full linting check passes
- Manual testing on demo, poetry, websocket modes
- All new effect plugins tested

## FILES CHANGED

- 24 files modified/added/deleted
- 723 insertions, 1,461 deletions (net -738 LOC - cleanup!)
- No breaking changes to public APIs
- All transitive imports updated correctly
2026-03-16 19:47:12 -07:00

MAINLINE

Digital consciousness stream. Matrix aesthetic · THX-1138 hue.

A full-screen terminal news ticker that renders live global headlines in large OTF-font block characters with a white-hot → deep green gradient. Headlines auto-translate into the native script of their subject region. Ambient mic input warps the glitch rate in real time. A --poetry mode replaces the feed with public-domain literary passages. Live messages can be pushed to the display over ntfy.sh.


Using

Run

python3 mainline.py                      # news stream
python3 mainline.py --poetry             # literary consciousness mode
python3 mainline.py -p                   # same
python3 mainline.py --firehose           # dense rapid-fire headline mode
python3 mainline.py --display websocket  # web browser display only
python3 mainline.py --display both       # terminal + web browser
python3 mainline.py --no-font-picker     # skip interactive font picker
python3 mainline.py --font-file path.otf # use a specific font file
python3 mainline.py --font-dir ~/fonts   # scan a different font folder
python3 mainline.py --font-index 1       # select face index within a collection

Or with uv:

uv run mainline.py

First run bootstraps dependencies. Use uv sync --all-extras for mic support.

Command & Control (C&C)

Control mainline remotely using cmdline.py:

uv run cmdline.py                    # Interactive TUI
uv run cmdline.py /effects list      # List all effects
uv run cmdline.py /effects stats     # Show performance stats
uv run cmdline.py -w /effects stats  # Watch mode (auto-refresh)

Commands are sent via ntfy.sh topics - useful for controlling a daemonized mainline instance.

Config

All constants live in engine/config.py:

Constant Default What it does
HEADLINE_LIMIT 1000 Total headlines per session
FEED_TIMEOUT 10 Per-feed HTTP timeout (seconds)
MIC_THRESHOLD_DB 50 dB floor above which glitches spike
NTFY_TOPIC klubhaus URL ntfy.sh JSON stream for messages
NTFY_CC_CMD_TOPIC klubhaus URL ntfy.sh topic for C&C commands
NTFY_CC_RESP_TOPIC klubhaus URL ntfy.sh topic for C&C responses
NTFY_RECONNECT_DELAY 5 Seconds before reconnecting after dropped SSE
MESSAGE_DISPLAY_SECS 30 How long an ntfy message holds the screen
FONT_DIR fonts/ Folder scanned for .otf, .ttf, .ttc files
FONT_PATH first file in FONT_DIR Active display font
FONT_PICKER True Show interactive font picker at boot
FONT_SZ 60 Font render size (affects block density)
RENDER_H 8 Terminal rows per headline line
SSAA 4 Super-sampling factor
SCROLL_DUR 5.625 Seconds per headline
FRAME_DT 0.05 Frame interval in seconds (20 FPS)
FIREHOSE_H 12 Firehose zone height (terminal rows)
GRAD_SPEED 0.08 Gradient sweep speed

Display Modes

Mainline supports multiple display backends:

  • Terminal (--display terminal): ANSI terminal output (default)
  • WebSocket (--display websocket): Stream to web browser clients
  • Sixel (--display sixel): Sixel graphics in supported terminals (iTerm2, mintty)
  • Both (--display both): Terminal + WebSocket simultaneously

WebSocket mode serves a web client at http://localhost:8766 with ANSI color support and fullscreen mode.

Feeds

~25 sources across four categories: Science & Technology, Economics & Business, World & Politics, Culture & Ideas. Add or swap feeds in engine/sources.pyFEEDS.

Poetry mode pulls from Project Gutenberg: Whitman, Dickinson, Thoreau, Emerson. Sources are in engine/sources.pyPOETRY_SOURCES.

Fonts

A fonts/ directory is bundled with demo faces. On startup, an interactive picker lists all discovered faces with a live half-block preview.

Navigation: / or j/k to move, Enter or q to select.

To add your own fonts, drop .otf, .ttf, or .ttc files into fonts/.

ntfy.sh

Mainline polls a configurable ntfy.sh topic in the background. When a message arrives, the scroll pauses and the message renders full-screen.

To push a message:

curl -d "Body text" -H "Title: Alert title" https://ntfy.sh/your_topic

Internals

How it works

  • On launch, the font picker scans fonts/ and presents a live-rendered TUI for face selection
  • Feeds are fetched and filtered on startup; results are cached for fast restarts
  • Headlines are rasterized via Pillow with 4× SSAA into half-block characters
  • The ticker uses a sweeping white-hot → deep green gradient
  • Subject-region detection triggers Google Translate and font swap for non-Latin scripts
  • The mic stream runs in a background thread, feeding RMS dB into glitch probability
  • The viewport scrolls through pre-rendered blocks with fade zones
  • An ntfy.sh SSE stream runs in a background thread for messages and C&C commands

Architecture

engine/
  __init__.py           package marker
  app.py                main(), font picker TUI, boot sequence, C&C poller
  config.py             constants, CLI flags, glyph tables
  sources.py            FEEDS, POETRY_SOURCES, language/script maps
  terminal.py           ANSI codes, tw/th, type_out, boot_ln
  filter.py             HTML stripping, content filter
  translate.py          Google Translate wrapper + region detection
  render.py             OTF → half-block pipeline (SSAA, gradient)
  effects/              plugin architecture for visual effects
    types.py            EffectPlugin ABC, EffectConfig, EffectContext
    registry.py         effect registration and lookup
    chain.py            effect pipeline chaining
    controller.py       handles /effects commands
    performance.py      performance monitoring
    legacy.py           legacy functional effects
  effects_plugins/      effect plugin implementations
    noise.py            noise effect
    fade.py             fade effect
    glitch.py           glitch effect
    firehose.py         firehose effect
  fetch.py              RSS/Gutenberg fetching + cache
  ntfy.py               NtfyPoller — standalone, zero internal deps
  mic.py                MicMonitor — standalone, graceful fallback
  scroll.py             stream() frame loop + message rendering
  viewport.py           terminal dimension tracking
  frame.py              scroll step calculation, timing
  layers.py             ticker zone, firehose, message overlay
  eventbus.py           thread-safe event publishing
  events.py             event types and definitions
  controller.py         coordinates ntfy/mic monitoring
  emitters.py           background emitters
  types.py              type definitions
  display/              Display backend system
    __init__.py         DisplayRegistry, get_monitor
    backends/
      terminal.py       ANSI terminal display
      websocket.py      WebSocket server for browser clients
      sixel.py         Sixel graphics (pure Python)
      null.py          headless display for testing
      multi.py         forwards to multiple displays
  benchmark.py          performance benchmarking tool

Development

Setup

Requires Python 3.10+ and uv.

uv sync                              # minimal (no mic)
uv sync --all-extras                 # with mic support
uv sync --all-extras --group dev     # full dev environment

Tasks

With mise:

mise run test            # run test suite
mise run test-cov       # run with coverage report

mise run lint           # ruff check
mise run lint-fix       # ruff check --fix
mise run format         # ruff format

mise run run            # terminal display
mise run run-websocket  # web display only
mise run run-sixel     # sixel graphics
mise run run-both       # terminal + web
mise run run-client     # both + open browser

mise run cmd            # C&C command interface
mise run cmd-stats      # watch effects stats

mise run benchmark      # run performance benchmarks
mise run benchmark-json # save as JSON

mise run topics-init    # initialize ntfy topics

Testing

uv run pytest
uv run pytest --cov=engine --cov-report=term-missing

# Run with mise
mise run test
mise run test-cov

# Run performance benchmarks
mise run benchmark
mise run benchmark-json

# Run benchmark hook mode (for CI)
uv run python -m engine.benchmark --hook

Performance regression tests are in tests/test_benchmark.py marked with @pytest.mark.benchmark.

Linting

uv run ruff check engine/ mainline.py
uv run ruff format engine/ mainline.py

Pre-commit hooks run lint automatically via hk.


Roadmap

Performance

  • Concurrent feed fetching with ThreadPoolExecutor
  • Background feed refresh daemon
  • Translation pre-fetch during boot

Graphics

  • Matrix rain katakana underlay
  • CRT scanline simulation
  • Sixel/iTerm2 inline images
  • Parallax secondary column

Cyberpunk Vibes

  • Keyword watch list with strobe effects
  • Breaking interrupt with synthesized audio
  • Live data overlay (BTC, ISS position)
  • Theme switcher (amber, ice, red)
  • Persona modes (surveillance, oracle, underground)

Python 3.10+. Primary display font is user-selectable via bundled fonts/ picker.

Description
A full-screen terminal news stream in a Matrix/THX-1138 aesthetic — scrolling live global headlines in large block type, with per-region translation and mic-reactive glitch effects. Supports a --poetry mode for public-domain literary passages.
Readme 4.6 MiB
Languages
Python 100%