Mainline/README.md

# MAINLINE

> *Digital consciousness stream. Matrix aesthetic · THX-1138 hue.*

A full-screen terminal news ticker that renders live global headlines in large OTF-font block characters with selectable color gradients (Verdant Green, Molten Orange, or Violet Purple). Headlines auto-translate into the native script of their subject region. Ambient mic input warps the glitch rate in real time. A `--poetry` mode replaces the feed with public-domain literary passages. Live messages can be pushed to the display over [ntfy.sh](https://ntfy.sh). **Figment mode** overlays flickery, theme-colored SVG glyphs on the running stream at timed intervals — controllable from any input source via an extensible trigger protocol.

---

## Contents

- [Using](#using)
  - [Run](#run)
  - [Config](#config)
  - [Display Modes](#display-modes)
  - [Feeds](#feeds)
  - [Fonts](#fonts)
  - [ntfy.sh](#ntfysh)
  - [Figment Mode](#figment-mode)
  - [Command & Control](#command--control-cc)
- [Internals](#internals)
  - [How it works](#how-it-works)
  - [Architecture](#architecture)
- [Development](#development)
  - [Setup](#setup)
  - [Tasks](#tasks)
  - [Testing](#testing)
  - [Linting](#linting)
- [Roadmap](#roadmap)
  - [Performance](#performance)
  - [Graphics](#graphics)
  - [Cyberpunk Vibes](#cyberpunk-vibes)
  - [Extensibility](#extensibility)

---

## Using

### Run

```bash
python3 mainline.py                      # news stream
python3 mainline.py --poetry             # literary consciousness mode
python3 mainline.py -p                   # same
python3 mainline.py --firehose           # dense rapid-fire headline mode
python3 mainline.py --figment            # enable periodic SVG glyph overlays
python3 mainline.py --figment-interval 30 # figment every 30 seconds (default: 60)
python3 mainline.py --display websocket  # web browser display only
python3 mainline.py --display both       # terminal + web browser
python3 mainline.py --refresh            # force re-fetch (bypass cache)
python3 mainline.py --no-font-picker     # skip interactive font picker
python3 mainline.py --font-file path.otf # use a specific font file
python3 mainline.py --font-dir ~/fonts   # scan a different font folder
python3 mainline.py --font-index 1       # select face index within a collection
```

Or with uv:

```bash
uv run mainline.py
```

First run bootstraps dependencies. Use `uv sync --all-extras` for mic support.

### Command & Control (C&C)

Control mainline remotely using `cmdline.py`:

```bash
uv run cmdline.py                    # Interactive TUI
uv run cmdline.py /effects list      # List all effects
uv run cmdline.py /effects stats     # Show performance stats
uv run cmdline.py -w /effects stats  # Watch mode (auto-refresh)
```

Commands are sent via ntfy.sh topics - useful for controlling a daemonized mainline instance.

### Config

All constants live in `engine/config.py`:

| Constant | Default | What it does |
|---|---|---|
| `HEADLINE_LIMIT` | `1000` | Total headlines per session |
| `FEED_TIMEOUT` | `10` | Per-feed HTTP timeout (seconds) |
| `MIC_THRESHOLD_DB` | `50` | dB floor above which glitches spike |
| `NTFY_TOPIC` | klubhaus URL | ntfy.sh JSON stream for messages |
| `NTFY_CC_CMD_TOPIC` | klubhaus URL | ntfy.sh topic for C&C commands |
| `NTFY_CC_RESP_TOPIC` | klubhaus URL | ntfy.sh topic for C&C responses |
| `NTFY_RECONNECT_DELAY` | `5` | Seconds before reconnecting after dropped SSE |
| `MESSAGE_DISPLAY_SECS` | `30` | How long an ntfy message holds the screen |
| `FONT_DIR` | `fonts/` | Folder scanned for `.otf`, `.ttf`, `.ttc` files |
| `FONT_PATH` | first file in `FONT_DIR` | Active display font |
| `FONT_PICKER` | `True` | Show interactive font picker at boot |
| `FONT_SZ` | `60` | Font render size (affects block density) |
| `RENDER_H` | `8` | Terminal rows per headline line |
| `SSAA` | `4` | Super-sampling factor |
| `SCROLL_DUR` | `5.625` | Seconds per headline |
| `FRAME_DT` | `0.05` | Frame interval in seconds (20 FPS) |
| `FIREHOSE_H` | `12` | Firehose zone height (terminal rows) |
| `GRAD_SPEED` | `0.08` | Gradient sweep speed |
| `FIGMENT_INTERVAL` | `60` | Seconds between figment appearances (set by `--figment-interval`) |

### Display Modes

Mainline supports multiple display backends:

- **Terminal** (`--display terminal`): ANSI terminal output (default)
- **WebSocket** (`--display websocket`): Stream to web browser clients
- **Sixel** (`--display sixel`): Sixel graphics in supported terminals (iTerm2, mintty)
- **Both** (`--display both`): Terminal + WebSocket simultaneously

WebSocket mode serves a web client at http://localhost:8766 with ANSI color support and fullscreen mode.

### Feeds

~25 sources across four categories: **Science & Technology**, **Economics & Business**, **World & Politics**, **Culture & Ideas**. Add or swap feeds in `engine/sources.py` → `FEEDS`.

**Poetry mode** pulls from Project Gutenberg: Whitman, Dickinson, Thoreau, Emerson. Sources are in `engine/sources.py` → `POETRY_SOURCES`.

### Fonts

A `fonts/` directory is bundled with demo faces. On startup, an interactive picker lists all discovered faces with a live half-block preview.

Navigation: `↑`/`↓` or `j`/`k` to move, `Enter` or `q` to select.

To add your own fonts, drop `.otf`, `.ttf`, or `.ttc` files into `fonts/`.

### ntfy.sh

Mainline polls a configurable ntfy.sh topic in the background. When a message arrives, the scroll pauses and the message renders full-screen.

To push a message:

```bash
curl -d "Body text" -H "Title: Alert title" https://ntfy.sh/your_topic
```

Update `NTFY_TOPIC` in `engine/config.py` to point at your own topic.

### Figment Mode

Figment mode periodically overlays a full-screen SVG glyph on the running ticker — flickering through a reveal → hold (strobe) → dissolve cycle, colored with a randomly selected theme gradient.

**Enable it** with the `--figment` flag:

```bash
uv run mainline.py --figment                        # glyph every 60 seconds (default)
uv run mainline.py --figment --figment-interval 30  # every 30 seconds
```

**Figment assets** live in `figments/` — drop any `.svg` file there and it will be picked up automatically. The bundled set contains Mayan and Aztec glyphs. Figments are selected randomly, avoiding immediate repeats, and rasterized into half-block terminal art at display time.

**Triggering manually** — any object with a `poll() -> FigmentCommand | None` method satisfies the `FigmentTrigger` protocol and can be passed to the plugin:

```python
from engine.figment_trigger import FigmentAction, FigmentCommand

class MyTrigger:
    def poll(self):
        if some_condition:
            return FigmentCommand(action=FigmentAction.TRIGGER)
        return None
```

Built-in commands: `TRIGGER`, `SET_INTENSITY`, `SET_INTERVAL`, `SET_COLOR`, `STOP`.

**System dependency:** Figment mode requires the Cairo C library (`brew install cairo` on macOS) in addition to the `figment` extras group:

```bash
uv sync --extra figment   # adds cairosvg
```

---

## Internals

### How it works

- On launch, the font picker scans `fonts/` and presents a live-rendered TUI for face selection; `--no-font-picker` skips directly to stream
- Feeds are fetched and filtered on startup (sports and vapid content stripped); results are cached to `.mainline_cache_news.json` / `.mainline_cache_poetry.json` for fast restarts
- Headlines are rasterized via Pillow with 4× SSAA into half-block characters (`▀▄█ `) at the configured font size
- The ticker uses a sweeping white-hot → deep green gradient; ntfy messages use a complementary white-hot → magenta/maroon gradient to distinguish them visually
- Subject-region detection runs a regex pass on each headline; matches trigger a Google Translate call and font swap to the appropriate script (CJK, Arabic, Devanagari, etc.) using macOS system fonts
- The mic stream runs in a background thread, feeding RMS dB into the glitch probability calculation each frame
- The viewport scrolls through a virtual canvas of pre-rendered blocks; fade zones at top and bottom dissolve characters probabilistically
- An ntfy.sh SSE stream runs in a background thread for messages and C&C commands; incoming messages interrupt the scroll and render full-screen until dismissed or expired
- Figment mode rasterizes SVGs via cairosvg → PIL → greyscale → half-block encode, then overlays them with ANSI cursor-positioning commands between the effect chain and the ntfy message layer

### Architecture

```
engine/
  __init__.py           package marker
  app.py                main(), font picker TUI, boot sequence, C&C poller
  config.py             constants, CLI flags, glyph tables
  sources.py            FEEDS, POETRY_SOURCES, language/script maps
  terminal.py           ANSI codes, tw/th, type_out, boot_ln
  filter.py             HTML stripping, content filter
  translate.py          Google Translate wrapper + region detection
  render.py             OTF → half-block pipeline (SSAA, gradient)
  effects/              plugin architecture for visual effects
    types.py            EffectPlugin ABC, EffectConfig, EffectContext
    registry.py         effect registration and lookup
    chain.py            effect pipeline chaining
    controller.py       handles /effects commands
    performance.py      performance monitoring
    legacy.py           legacy functional effects
  fetch.py              RSS/Gutenberg fetching + cache load/save
  ntfy.py               NtfyPoller — standalone, zero internal deps
  mic.py                MicMonitor — standalone, graceful fallback
  scroll.py             stream() frame loop + message rendering
  viewport.py           terminal dimension tracking (tw/th)
  frame.py              scroll step calculation, timing
  layers.py             ticker zone, firehose, message + figment overlay rendering
  figment_render.py     SVG → cairosvg → PIL → half-block rasterizer with cache
  figment_trigger.py    FigmentTrigger protocol, FigmentAction enum, FigmentCommand
  eventbus.py           thread-safe event publishing for decoupled communication
  events.py             event types and definitions
  controller.py         coordinates ntfy/mic monitoring and event publishing
  emitters.py           background emitters for ntfy and mic
  types.py              type definitions and dataclasses
  themes.py             THEME_REGISTRY — gradient color definitions
  display/              Display backend system
    __init__.py         DisplayRegistry, get_monitor
    backends/
      terminal.py       ANSI terminal display
      websocket.py      WebSocket server for browser clients
      sixel.py          Sixel graphics (pure Python)
      null.py           headless display for testing
      multi.py          forwards to multiple displays
  benchmark.py          performance benchmarking tool

effects_plugins/
  __init__.py           plugin discovery (ABC issubclass scan)
  noise.py              NoiseEffect — random character noise
  glitch.py             GlitchEffect — horizontal glitch bars
  fade.py               FadeEffect — edge fade zones
  firehose.py           FirehoseEffect — dense bottom ticker strip
  figment.py            FigmentEffect — periodic SVG glyph overlay (state machine)

figments/               SVG assets for figment mode
```

---

## Development

### Setup

Requires Python 3.10+ and [uv](https://docs.astral.sh/uv/).

```bash
uv sync                              # minimal (no mic, no figment)
uv sync --extra mic                  # with mic support (sounddevice + numpy)
uv sync --extra figment              # with figment mode (cairosvg + system Cairo)
uv sync --all-extras                 # all optional features
uv sync --all-extras --group dev     # full dev environment
```

Figment mode also requires the Cairo C library: `brew install cairo` (macOS).

### Tasks

With [mise](https://mise.jdx.dev/):

```bash
mise run test            # run test suite
mise run test-cov       # run with coverage report

mise run lint           # ruff check
mise run lint-fix       # ruff check --fix
mise run format         # ruff format

mise run run            # terminal display
mise run run-websocket  # web display only
mise run run-sixel     # sixel graphics
mise run run-both       # terminal + web
mise run run-client     # both + open browser

mise run cmd            # C&C command interface
mise run cmd-stats      # watch effects stats

mise run benchmark      # run performance benchmarks
mise run benchmark-json # save as JSON

mise run topics-init    # initialize ntfy topics
```

### Testing

Tests live in `tests/` and cover `config`, `filter`, `mic`, `ntfy`, `sources`, `terminal`, and the full figment pipeline (`figment_render`, `figment_trigger`, `figment`, `figment_overlay`). Figment tests are automatically skipped if Cairo is not installed.

```bash
uv run pytest
uv run pytest --cov=engine --cov-report=term-missing

# Run with mise
mise run test
mise run test-cov

# Run performance benchmarks
mise run benchmark
mise run benchmark-json

# Run benchmark hook mode (for CI)
uv run python -m engine.benchmark --hook
```

Performance regression tests are in `tests/test_benchmark.py` marked with `@pytest.mark.benchmark`.

### Linting

```bash
uv run ruff check engine/ mainline.py
uv run ruff format engine/ mainline.py
```

Pre-commit hooks run lint automatically via `hk`.

---

## Roadmap

### Performance
- Concurrent feed fetching with ThreadPoolExecutor
- Background feed refresh daemon
- Translation pre-fetch during boot

### Graphics
- Matrix rain katakana underlay
- CRT scanline simulation
- Sixel/iTerm2 inline images
- Parallax secondary column

### Cyberpunk Vibes
- **Figment intensity wiring** — `config.intensity` currently stored but not yet applied to reveal/dissolve speed or strobe frequency
- **ntfy figment trigger** — built-in `NtfyFigmentTrigger` that listens on a dedicated topic to fire figments on demand
- **Keyword watch list** — highlight or strobe any headline matching tracked terms (names, topics, tickers)
- **Breaking interrupt** — full-screen flash + synthesized blip when a high-priority keyword hits
- **Live data overlay** — secondary ticker strip at screen edge: BTC price, ISS position, geomagnetic index
- **Theme switcher** — `--amber` (phosphor), `--ice` (electric cyan), `--red` (alert state) palette modes via CLI flag
- **Persona modes** — `--surveillance`, `--oracle`, `--underground` as feed presets with matching color themes and boot copy
- **Synthesized audio** — short static bursts tied to glitch events, independent of mic input

### Extensibility
- **serve.py** — HTTP server that imports `engine.render` and `engine.fetch` directly to stream 1-bit bitmaps to an ESP32 display
- **Rust port** — `ntfy.py` and `render.py` are the natural first targets; clear module boundaries make incremental porting viable

---

*Python 3.10+. Primary display font is user-selectable via bundled `fonts/` picker.*