David Gwilliam 2c23c423a0 feat(hybrid): Add hybrid preset-graph configuration system
Implement Option 5: Hybrid preset-graph system that combines preset
simplicity with graph flexibility, providing 70% reduction in config
file size compared to verbose node DSL.

## New Files

- engine/pipeline/hybrid_config.py - Core hybrid config parser
- examples/hybrid_config.toml - Example hybrid configuration (20 lines)
- examples/hybrid_visualization.py - Demo script using hybrid config
- tests/test_hybrid_config.py - Comprehensive test suite (17 tests)
- docs/hybrid-config.md - Complete documentation

## Key Features

1. **Concise Syntax** (70% smaller than verbose DSL):

2. **Automatic Connections**: Linear pipeline order is inferred

3. **Flexible Configuration**:
   - Inline objects:
   - Array notation:
   - Shorthand:

4. **Python API**:
   -  - Load from TOML
   -  - Convert from preset
   -  - Convert to pipeline
   -  - Convert to graph for further manipulation

## Usage

Loading hybrid configuration...
======================================================================
✓ Hybrid config loaded from hybrid_config.toml
  Source: headlines
  Camera: scroll
  Effects: 4
    - noise: intensity=0.3
    - fade: intensity=0.5
    - glitch: intensity=0.2
    - firehose: intensity=0.4
  Display: terminal
  Auto-injected stages for missing capabilities: ['camera_update', 'render']
✓ Pipeline created with 9 stages
  Stages: ['source', 'camera', 'noise', 'fade', 'glitch', 'firehose', 'display', 'camera_update', 'render']
[?25l✓ Pipeline initialized
Executing pipeline...
  > MIT Tech Review .............................. LINKED [10]
  > Quanta .............................. LINKED [5]
  > Phys.org .............................. LINKED [30]
  > Ars Technica .............................. LINKED [20]
  > Science Daily .............................. LINKED [60]
  > Nature .............................. LINKED [75]
  > New Scientist .............................. LINKED [99]
  > NASA .............................. LINKED [10]
  > BBC Business .............................. LINKED [54]
  > BBC Science .............................. LINKED [36]
  > MarketWatch .............................. LINKED [10]
  > NPR .............................. LINKED [10]
  > Economist .............................. LINKED [299]
  > Al Jazeera .............................. LINKED [25]
  > France24 .............................. LINKED [24]
  > Guardian World .............................. LINKED [45]
  > BBC World .............................. LINKED [28]
  > ABC Australia .............................. LINKED [23]
  > DW .............................. LINKED [124]
  > Smithsonian .............................. LINKED [10]
  > Aeon .............................. LINKED [20]
  > Wired .............................. LINKED [48]
  > The Hindu .............................. LINKED [60]
  > Japan Times .............................. LINKED [29]
  > Nautilus .............................. LINKED [10]
  > Guardian Culture .............................. LINKED [24]
  > Literary Hub .............................. LINKED [10]
  > The Conversation .............................. LINKED [48]
  > The Marginalian .............................. LINKED [20]
  > Longreads .............................. LINKED [25]
  > Der Spiegel .............................. LINKED [19]
  > Atlas Obscura .............................. LINKED [27]
  > SCMP ..............................The Download: OpenAI is building a fully automated researcher, and a psychedelic
 pe                e  r          o      in                     e  a
    -      n    b  an          t        l       r                i l
       nl ad     n    co  ut n      h  l h  a    h  t e  o  d d     t r   c e
C n  ua t m co    e s             a  h  a e p      s          o  f nd
     h  w r    o  n    ec  le  o e   cl  r  a  e
T e D w  o     h   en a o ’s new A     ns, and n x -  n  u   a  r  c   s
W  t do ne  nucl ar r   tors  ea  f   w s  ?
 h  Penta o   s  l nni g  or  I co p nies  o tr in    cl s   i d   t   def nse o
T    ownl  d   pe  I s  S mi  t    dea , an    ok’  CS M   ws it
T   J  lies T a   vol  d       er nt   y    K     i e
Qu nt m   y  o  ap   Pi  ee     n   r  g    rd
T e  a h T a  E p  i     y B     urve  Are  ver   er
Why     u a   d        Stil   t u  le W t  t      ll S uff?
W e e  ome  ee S     s, She S es   S ace T  e M  e o  F ac  l
      ウ┋          ウ ホ          ウ ┆            メ   キ          ケ ┃            
Ligh -  s d     n  u      t s ar  f cia  str    r     a    mi   h  s   f    ng o
New resea  h exp  r s  h   a ad   of  i ms' u  q e t chnol g  s
L mi e  j    bl  k  oc     ob      o por  nit  s f   y  ng pe     in  oas  l  n
Are hu a    a ural   vi l nt?  ew re  arc  c  ll       o  - e   a s     ons
 a     m l      e e r  q a  s?
New      cove e  p o  s   ow  stro      eil   m t     a ter t   Ge in  8 e e
 o          a t                 g   3     a    g ye    b        r             b
How DICER cuts microRNAs with single-nucleotide precision                        LINKED [50]
======================================================================
Visualization Output:
======================================================================
The Download: OpenAI is building a fully automated researcher, and a psychedelic
 pe                e  r          o      in                     e  a
    -      n    b  an          t        l       r                i l
       nl ad     n    co  ut n      h  l h  a    h  t e  o  d d     t r   c e
C n  ua t m co    e s             a  h  a e p      s          o  f nd
     h  w r    o  n    ec  le  o e   cl  r  a  e
T e D w  o     h   en a o ’s new A     ns, and n x -  n  u   a  r  c   s
W  t do ne  nucl ar r   tors  ea  f   w s  ?
 h  Penta o   s  l nni g  or  I co p nies  o tr in    cl s   i d   t   def nse o
T    ownl  d   pe  I s  S mi  t    dea , an    ok’  CS M   ws it
T   J  lies T a   vol  d       er nt   y    K     i e
Qu nt m   y  o  ap   Pi  ee     n   r  g    rd
T e  a h T a  E p  i     y B     urve  Are  ver   er
Why     u a   d        Stil   t u  le W t  t      ll S uff?
W e e  ome  ee S     s, She S es   S ace T  e M  e o  F ac  l
      ウ┋          ウ ホ          ウ ┆            メ   キ          ケ ┃            
Ligh -  s d     n  u      t s ar  f cia  str    r     a    mi   h  s   f    ng o
New resea  h exp  r s  h   a ad   of  i ms' u  q e t chnol g  s
L mi e  j    bl  k  oc     ob      o por  nit  s f   y  ng pe     in  oas  l  n
Are hu a    a ural   vi l nt?  ew re  arc  c  ll       o  - e   a s     ons
 a     m l      e e r  q a  s?
New      cove e  p o  s   ow  stro      eil   m t     a ter t   Ge in  8 e e
 o          a t                 g   3     a    g ye    b        r             b
How DICER cuts microRNAs with single-nucleotide precision
======================================================================
✓ Successfully rendered 24 lines

## Comparison

| Format | Lines | Use Case |
|--------|-------|----------|
| Preset | 10 | Simple configs |
| **Hybrid** | **20** | **Most use cases (recommended)** |
| Verbose DSL | 39 | Complex DAGs |

All existing functionality preserved - verbose node DSL still works.
2026-03-21 21:03:27 -07:00
2026-03-21 19:27:06 -07:00

MAINLINE

Digital consciousness stream. Matrix aesthetic · THX-1138 hue.

A full-screen terminal news ticker that renders live global headlines in large OTF-font block characters with a white-hot → deep green gradient. Headlines auto-translate into the native script of their subject region. Ambient mic input warps the glitch rate in real time. A --poetry mode replaces the feed with public-domain literary passages. Live messages can be pushed to the display over ntfy.sh.


Using

Run

python3 mainline.py                      # news stream
python3 mainline.py --poetry             # literary consciousness mode
python3 mainline.py -p                   # same
python3 mainline.py --firehose           # dense rapid-fire headline mode
python3 mainline.py --display websocket  # web browser display only
python3 mainline.py --no-font-picker     # skip interactive font picker
python3 mainline.py --font-file path.otf # use a specific font file
python3 mainline.py --font-dir ~/fonts   # scan a different font folder
python3 mainline.py --font-index 1       # select face index within a collection

Or with uv:

uv run mainline.py

First run bootstraps dependencies. Use uv sync --all-extras for mic support.

Command & Control (C&C)

Control mainline remotely using cmdline.py:

uv run cmdline.py                    # Interactive TUI
uv run cmdline.py /effects list      # List all effects
uv run cmdline.py /effects stats     # Show performance stats
uv run cmdline.py -w /effects stats  # Watch mode (auto-refresh)

Commands are sent via ntfy.sh topics - useful for controlling a daemonized mainline instance.

Config

All constants live in engine/config.py:

Constant Default What it does
HEADLINE_LIMIT 1000 Total headlines per session
FEED_TIMEOUT 10 Per-feed HTTP timeout (seconds)
MIC_THRESHOLD_DB 50 dB floor above which glitches spike
NTFY_TOPIC klubhaus URL ntfy.sh JSON stream for messages
NTFY_CC_CMD_TOPIC klubhaus URL ntfy.sh topic for C&C commands
NTFY_CC_RESP_TOPIC klubhaus URL ntfy.sh topic for C&C responses
NTFY_RECONNECT_DELAY 5 Seconds before reconnecting after dropped SSE
MESSAGE_DISPLAY_SECS 30 How long an ntfy message holds the screen
FONT_DIR fonts/ Folder scanned for .otf, .ttf, .ttc files
FONT_PATH first file in FONT_DIR Active display font
FONT_PICKER True Show interactive font picker at boot
FONT_SZ 60 Font render size (affects block density)
RENDER_H 8 Terminal rows per headline line
SSAA 4 Super-sampling factor
SCROLL_DUR 5.625 Seconds per headline
FRAME_DT 0.05 Frame interval in seconds (20 FPS)
FIREHOSE_H 12 Firehose zone height (terminal rows)
GRAD_SPEED 0.08 Gradient sweep speed

Display Modes

Mainline supports multiple display backends:

  • Terminal (--display terminal): ANSI terminal output (default)
  • WebSocket (--display websocket): Stream to web browser clients
  • ModernGL (--display moderngl): GPU-accelerated rendering (optional)

WebSocket mode serves a web client at http://localhost:8766 with ANSI color support and fullscreen mode.

Feeds

~25 sources across four categories: Science & Technology, Economics & Business, World & Politics, Culture & Ideas. Add or swap feeds in engine/sources.pyFEEDS.

Poetry mode pulls from Project Gutenberg: Whitman, Dickinson, Thoreau, Emerson. Sources are in engine/sources.pyPOETRY_SOURCES.

Fonts

A fonts/ directory is bundled with demo faces. On startup, an interactive picker lists all discovered faces with a live half-block preview.

Navigation: / or j/k to move, Enter or q to select.

To add your own fonts, drop .otf, .ttf, or .ttc files into fonts/.

ntfy.sh

Mainline polls a configurable ntfy.sh topic in the background. When a message arrives, the scroll pauses and the message renders full-screen.

To push a message:

curl -d "Body text" -H "Title: Alert title" https://ntfy.sh/your_topic

Internals

How it works

  • On launch, the font picker scans fonts/ and presents a live-rendered TUI for face selection
  • Feeds are fetched and filtered on startup; results are cached for fast restarts
  • Headlines are rasterized via Pillow with 4× SSAA into half-block characters
  • The ticker uses a sweeping white-hot → deep green gradient
  • Subject-region detection triggers Google Translate and font swap for non-Latin scripts
  • The mic stream runs in a background thread, feeding RMS dB into glitch probability
  • The viewport scrolls through pre-rendered blocks with fade zones
  • An ntfy.sh SSE stream runs in a background thread for messages and C&C commands

Architecture

engine/
  __init__.py           package marker
  app.py                main(), font picker TUI, boot sequence, C&C poller
  config.py             constants, CLI flags, glyph tables
  sources.py            FEEDS, POETRY_SOURCES, language/script maps
  terminal.py           ANSI codes, tw/th, type_out, boot_ln
  filter.py             HTML stripping, content filter
  translate.py          Google Translate wrapper + region detection
  render.py             OTF → half-block pipeline (SSAA, gradient)
  effects/              plugin architecture for visual effects
    types.py            EffectPlugin ABC, EffectConfig, EffectContext
    registry.py         effect registration and lookup
    chain.py            effect pipeline chaining
    controller.py       handles /effects commands
    performance.py      performance monitoring
    legacy.py           legacy functional effects
  effects_plugins/      effect plugin implementations
    noise.py            noise effect
    fade.py             fade effect
    glitch.py           glitch effect
    firehose.py         firehose effect
  fetch.py              RSS/Gutenberg fetching + cache
  ntfy.py               NtfyPoller — standalone, zero internal deps
  mic.py                MicMonitor — standalone, graceful fallback
  scroll.py             stream() frame loop + message rendering
  viewport.py           terminal dimension tracking
  frame.py              scroll step calculation, timing
  layers.py             ticker zone, firehose, message overlay
  eventbus.py           thread-safe event publishing
  events.py             event types and definitions
  controller.py         coordinates ntfy/mic monitoring
  emitters.py           background emitters
  types.py              type definitions
  display/              Display backend system
    __init__.py         DisplayRegistry, get_monitor
    backends/
      terminal.py       ANSI terminal display
      websocket.py      WebSocket server for browser clients
      null.py          headless display for testing
      multi.py         forwards to multiple displays
      moderngl.py       GPU-accelerated OpenGL rendering
  benchmark.py          performance benchmarking tool

Development

Setup

Requires Python 3.10+ and uv.

uv sync                              # minimal (no mic)
uv sync --all-extras                 # with mic support
uv sync --all-extras --group dev     # full dev environment

Tasks

With mise:

mise run test            # run test suite
mise run test-cov       # run with coverage report

mise run lint           # ruff check
mise run lint-fix       # ruff check --fix
mise run format         # ruff format

mise run run            # terminal display
mise run run-websocket  # web display only
mise run run-client     # terminal + web

mise run cmd            # C&C command interface
mise run cmd-stats      # watch effects stats

mise run benchmark      # run performance benchmarks
mise run benchmark-json # save as JSON

mise run topics-init    # initialize ntfy topics

Testing

uv run pytest
uv run pytest --cov=engine --cov-report=term-missing

# Run with mise
mise run test
mise run test-cov

# Run performance benchmarks
mise run benchmark
mise run benchmark-json

# Run benchmark hook mode (for CI)
uv run python -m engine.benchmark --hook

Performance regression tests are in tests/test_benchmark.py marked with @pytest.mark.benchmark.

Linting

uv run ruff check engine/ mainline.py
uv run ruff format engine/ mainline.py

Pre-commit hooks run lint automatically via hk.


Roadmap

Performance

  • Concurrent feed fetching with ThreadPoolExecutor
  • Background feed refresh daemon
  • Translation pre-fetch during boot

Graphics

  • Matrix rain katakana underlay
  • CRT scanline simulation
  • Sixel/iTerm2 inline images
  • Parallax secondary column

Cyberpunk Vibes

  • Keyword watch list with strobe effects
  • Breaking interrupt with synthesized audio
  • Live data overlay (BTC, ISS position)
  • Theme switcher (amber, ice, red)
  • Persona modes (surveillance, oracle, underground)

Python 3.10+. Primary display font is user-selectable via bundled fonts/ picker.

Description
A full-screen terminal news stream in a Matrix/THX-1138 aesthetic — scrolling live global headlines in large block type, with per-region translation and mic-reactive glitch effects. Supports a --poetry mode for public-domain literary passages.
Readme 4.6 MiB
Languages
Python 100%