## Summary Fixed critical performance issue where demo/poetry presets would hang for 10+ seconds due to FontStage rendering all 1438+ headline items instead of just the visible ~5 items. ## Changes ### Core Fix: ViewportFilterStage - New pipeline stage that filters items to only those fitting in the viewport - Reduces 1438 items → ~5 items (288x reduction) before FontStage - Prevents expensive PIL font rendering operations on items that won't be displayed - Located: engine/pipeline/adapters.py:348-403 ### Pipeline Integration - Updated app.py to add ViewportFilterStage before FontStage for headlines/poetry sources - Ensures correct data flow: source → viewport_filter → font → camera → effects → display - ViewportFilterStage depends on 'source' capability, providing pass-through filtering ### Display Protocol Enhancement - Added is_quit_requested() and clear_quit_request() method signatures to Display protocol - Documented as optional methods for backends supporting keyboard input - Already implemented by pygame backend, now formally part of protocol ### Debug Infrastructure - Added MAINLINE_DEBUG_DATAFLOW environment variable logging throughout pipeline - Logs stage input/output types and data sizes to stderr (when flag enabled) - Verified working: 1438 → 5 item reduction shown in debug output ### Performance Testing - Added pytest-benchmark (v5.2.3) as dev dependency for statistical benchmarking - Created comprehensive performance regression tests (tests/test_performance_regression.py) - Tests verify: - ViewportFilterStage filters 2000 items efficiently (<1ms) - FontStage processes filtered items quickly (<50ms) - 288x performance improvement ratio maintained - Pipeline doesn't hang with large datasets - All 523 tests passing, including 7 new performance tests ## Performance Impact **Before:** FontStage renders all 1438 items per frame → 10+ second hang **After:** FontStage renders ~5 items per frame → sub-second execution Real-world impact: Demo preset now responsive and usable with news sources. ## Testing - Unit tests: 523 passed, 16 skipped - Regression tests: Catch performance degradation with large datasets - E2E verification: Debug logging confirms correct pipeline flow - Benchmark suite: Statistical performance tracking enabled
MAINLINE
Digital consciousness stream. Matrix aesthetic · THX-1138 hue.
A full-screen terminal news ticker that renders live global headlines in large OTF-font block characters with a white-hot → deep green gradient. Headlines auto-translate into the native script of their subject region. Ambient mic input warps the glitch rate in real time. A --poetry mode replaces the feed with public-domain literary passages. Live messages can be pushed to the display over ntfy.sh.
Using
Run
python3 mainline.py # news stream
python3 mainline.py --poetry # literary consciousness mode
python3 mainline.py -p # same
python3 mainline.py --firehose # dense rapid-fire headline mode
python3 mainline.py --display websocket # web browser display only
python3 mainline.py --display both # terminal + web browser
python3 mainline.py --no-font-picker # skip interactive font picker
python3 mainline.py --font-file path.otf # use a specific font file
python3 mainline.py --font-dir ~/fonts # scan a different font folder
python3 mainline.py --font-index 1 # select face index within a collection
Or with uv:
uv run mainline.py
First run bootstraps dependencies. Use uv sync --all-extras for mic support.
Command & Control (C&C)
Control mainline remotely using cmdline.py:
uv run cmdline.py # Interactive TUI
uv run cmdline.py /effects list # List all effects
uv run cmdline.py /effects stats # Show performance stats
uv run cmdline.py -w /effects stats # Watch mode (auto-refresh)
Commands are sent via ntfy.sh topics - useful for controlling a daemonized mainline instance.
Config
All constants live in engine/config.py:
| Constant | Default | What it does |
|---|---|---|
HEADLINE_LIMIT |
1000 |
Total headlines per session |
FEED_TIMEOUT |
10 |
Per-feed HTTP timeout (seconds) |
MIC_THRESHOLD_DB |
50 |
dB floor above which glitches spike |
NTFY_TOPIC |
klubhaus URL | ntfy.sh JSON stream for messages |
NTFY_CC_CMD_TOPIC |
klubhaus URL | ntfy.sh topic for C&C commands |
NTFY_CC_RESP_TOPIC |
klubhaus URL | ntfy.sh topic for C&C responses |
NTFY_RECONNECT_DELAY |
5 |
Seconds before reconnecting after dropped SSE |
MESSAGE_DISPLAY_SECS |
30 |
How long an ntfy message holds the screen |
FONT_DIR |
fonts/ |
Folder scanned for .otf, .ttf, .ttc files |
FONT_PATH |
first file in FONT_DIR |
Active display font |
FONT_PICKER |
True |
Show interactive font picker at boot |
FONT_SZ |
60 |
Font render size (affects block density) |
RENDER_H |
8 |
Terminal rows per headline line |
SSAA |
4 |
Super-sampling factor |
SCROLL_DUR |
5.625 |
Seconds per headline |
FRAME_DT |
0.05 |
Frame interval in seconds (20 FPS) |
FIREHOSE_H |
12 |
Firehose zone height (terminal rows) |
GRAD_SPEED |
0.08 |
Gradient sweep speed |
Display Modes
Mainline supports multiple display backends:
- Terminal (
--display terminal): ANSI terminal output (default) - WebSocket (
--display websocket): Stream to web browser clients - Sixel (
--display sixel): Sixel graphics in supported terminals (iTerm2, mintty) - Both (
--display both): Terminal + WebSocket simultaneously
WebSocket mode serves a web client at http://localhost:8766 with ANSI color support and fullscreen mode.
Feeds
~25 sources across four categories: Science & Technology, Economics & Business, World & Politics, Culture & Ideas. Add or swap feeds in engine/sources.py → FEEDS.
Poetry mode pulls from Project Gutenberg: Whitman, Dickinson, Thoreau, Emerson. Sources are in engine/sources.py → POETRY_SOURCES.
Fonts
A fonts/ directory is bundled with demo faces. On startup, an interactive picker lists all discovered faces with a live half-block preview.
Navigation: ↑/↓ or j/k to move, Enter or q to select.
To add your own fonts, drop .otf, .ttf, or .ttc files into fonts/.
ntfy.sh
Mainline polls a configurable ntfy.sh topic in the background. When a message arrives, the scroll pauses and the message renders full-screen.
To push a message:
curl -d "Body text" -H "Title: Alert title" https://ntfy.sh/your_topic
Internals
How it works
- On launch, the font picker scans
fonts/and presents a live-rendered TUI for face selection - Feeds are fetched and filtered on startup; results are cached for fast restarts
- Headlines are rasterized via Pillow with 4× SSAA into half-block characters
- The ticker uses a sweeping white-hot → deep green gradient
- Subject-region detection triggers Google Translate and font swap for non-Latin scripts
- The mic stream runs in a background thread, feeding RMS dB into glitch probability
- The viewport scrolls through pre-rendered blocks with fade zones
- An ntfy.sh SSE stream runs in a background thread for messages and C&C commands
Architecture
engine/
__init__.py package marker
app.py main(), font picker TUI, boot sequence, C&C poller
config.py constants, CLI flags, glyph tables
sources.py FEEDS, POETRY_SOURCES, language/script maps
terminal.py ANSI codes, tw/th, type_out, boot_ln
filter.py HTML stripping, content filter
translate.py Google Translate wrapper + region detection
render.py OTF → half-block pipeline (SSAA, gradient)
effects/ plugin architecture for visual effects
types.py EffectPlugin ABC, EffectConfig, EffectContext
registry.py effect registration and lookup
chain.py effect pipeline chaining
controller.py handles /effects commands
performance.py performance monitoring
legacy.py legacy functional effects
effects_plugins/ effect plugin implementations
noise.py noise effect
fade.py fade effect
glitch.py glitch effect
firehose.py firehose effect
fetch.py RSS/Gutenberg fetching + cache
ntfy.py NtfyPoller — standalone, zero internal deps
mic.py MicMonitor — standalone, graceful fallback
scroll.py stream() frame loop + message rendering
viewport.py terminal dimension tracking
frame.py scroll step calculation, timing
layers.py ticker zone, firehose, message overlay
eventbus.py thread-safe event publishing
events.py event types and definitions
controller.py coordinates ntfy/mic monitoring
emitters.py background emitters
types.py type definitions
display/ Display backend system
__init__.py DisplayRegistry, get_monitor
backends/
terminal.py ANSI terminal display
websocket.py WebSocket server for browser clients
sixel.py Sixel graphics (pure Python)
null.py headless display for testing
multi.py forwards to multiple displays
benchmark.py performance benchmarking tool
Development
Setup
Requires Python 3.10+ and uv.
uv sync # minimal (no mic)
uv sync --all-extras # with mic support
uv sync --all-extras --group dev # full dev environment
Tasks
With mise:
mise run test # run test suite
mise run test-cov # run with coverage report
mise run lint # ruff check
mise run lint-fix # ruff check --fix
mise run format # ruff format
mise run run # terminal display
mise run run-websocket # web display only
mise run run-sixel # sixel graphics
mise run run-both # terminal + web
mise run run-client # both + open browser
mise run cmd # C&C command interface
mise run cmd-stats # watch effects stats
mise run benchmark # run performance benchmarks
mise run benchmark-json # save as JSON
mise run topics-init # initialize ntfy topics
Testing
uv run pytest
uv run pytest --cov=engine --cov-report=term-missing
# Run with mise
mise run test
mise run test-cov
# Run performance benchmarks
mise run benchmark
mise run benchmark-json
# Run benchmark hook mode (for CI)
uv run python -m engine.benchmark --hook
Performance regression tests are in tests/test_benchmark.py marked with @pytest.mark.benchmark.
Linting
uv run ruff check engine/ mainline.py
uv run ruff format engine/ mainline.py
Pre-commit hooks run lint automatically via hk.
Roadmap
Performance
- Concurrent feed fetching with ThreadPoolExecutor
- Background feed refresh daemon
- Translation pre-fetch during boot
Graphics
- Matrix rain katakana underlay
- CRT scanline simulation
- Sixel/iTerm2 inline images
- Parallax secondary column
Cyberpunk Vibes
- Keyword watch list with strobe effects
- Breaking interrupt with synthesized audio
- Live data overlay (BTC, ISS position)
- Theme switcher (amber, ice, red)
- Persona modes (surveillance, oracle, underground)
Python 3.10+. Primary display font is user-selectable via bundled fonts/ picker.