- Rename VERTICAL camera mode to FEED (rapid single-item view)
- Add SCROLL camera mode with float accumulation for smooth movie-credits style scrolling
- Add estimate_block_height() for cheap layout calculation without full rendering
- Replace ViewportFilterStage with layout-aware filtering that tracks camera position
- Add render caching to FontStage to avoid re-rendering items
- Fix CameraStage to use global canvas height for scrolling bounds
- Add horizontal padding in Camera.apply() to prevent ghosting
- Add get_dimensions() to MultiDisplay for proper viewport sizing
- Fix PygameDisplay to auto-detect viewport from window size
- Update presets to use scroll camera with appropriate speeds
## Summary
Fixed critical performance issue where demo/poetry presets would hang for 10+ seconds due to FontStage rendering all 1438+ headline items instead of just the visible ~5 items.
## Changes
### Core Fix: ViewportFilterStage
- New pipeline stage that filters items to only those fitting in the viewport
- Reduces 1438 items → ~5 items (288x reduction) before FontStage
- Prevents expensive PIL font rendering operations on items that won't be displayed
- Located: engine/pipeline/adapters.py:348-403
### Pipeline Integration
- Updated app.py to add ViewportFilterStage before FontStage for headlines/poetry sources
- Ensures correct data flow: source → viewport_filter → font → camera → effects → display
- ViewportFilterStage depends on 'source' capability, providing pass-through filtering
### Display Protocol Enhancement
- Added is_quit_requested() and clear_quit_request() method signatures to Display protocol
- Documented as optional methods for backends supporting keyboard input
- Already implemented by pygame backend, now formally part of protocol
### Debug Infrastructure
- Added MAINLINE_DEBUG_DATAFLOW environment variable logging throughout pipeline
- Logs stage input/output types and data sizes to stderr (when flag enabled)
- Verified working: 1438 → 5 item reduction shown in debug output
### Performance Testing
- Added pytest-benchmark (v5.2.3) as dev dependency for statistical benchmarking
- Created comprehensive performance regression tests (tests/test_performance_regression.py)
- Tests verify:
- ViewportFilterStage filters 2000 items efficiently (<1ms)
- FontStage processes filtered items quickly (<50ms)
- 288x performance improvement ratio maintained
- Pipeline doesn't hang with large datasets
- All 523 tests passing, including 7 new performance tests
## Performance Impact
**Before:** FontStage renders all 1438 items per frame → 10+ second hang
**After:** FontStage renders ~5 items per frame → sub-second execution
Real-world impact: Demo preset now responsive and usable with news sources.
## Testing
- Unit tests: 523 passed, 16 skipped
- Regression tests: Catch performance degradation with large datasets
- E2E verification: Debug logging confirms correct pipeline flow
- Benchmark suite: Statistical performance tracking enabled
Two critical fixes:
1. ListDataSource Cache Bug
- Previously, ListDataSource.__init__ cached raw tuples directly
- get_items() would return cached raw tuples without converting to SourceItem
- This caused SourceItemsToBufferStage to receive tuples and stringify them
- Results: ugly tuple representations in terminal/pygame instead of formatted text
- Fix: Store raw items in _raw_items, let fetch() convert to SourceItem
- Cache now contains proper SourceItem objects
2. Camera Dependency Resolution
- CameraStage declared dependency on 'source.items' exactly
- DataSourceStage provides 'source.headlines' (or 'source.poetry', etc.)
- Capability matching didn't trigger prefix match for exact dependency
- Fix: Change CameraStage dependency to 'source' for prefix matching
3. Added app.py Camera Stage Support
- Pipeline now adds camera stage from preset.camera config
- Supports vertical, horizontal, omni, floating, bounce modes
- Tests now passing with proper data flow through all stages
Tests: All 502 tests passing, 16 skipped
- Remove RenderStage import from engine/app.py
- Replace RenderStage with SourceItemsToBufferStage for all sources
- Simplifies render pipeline - no more special-case logic
- SourceItemsToBufferStage properly converts items to text buffer
- Tests pass (11 app tests)
Move internal imports in run_pipeline_mode() to module level to support
proper mocking in integration tests. This enables more effective testing
of the app's initialization and pipeline setup.
Also simplifies the test suite to focus on key integration points.
Changes:
- Moved effects_plugins, DisplayRegistry, PerformanceMonitor, fetch functions,
and pipeline adapters to module-level imports
- Removed duplicate imports from run_pipeline_mode()
- Simplified test_app.py to focus on core functionality
All manual tests still pass (border-test preset works correctly).
The --display CLI flag wasn't being checked, so it was always using
the preset's display backend.
Now app.py checks if --display was provided and uses it if present,
otherwise falls back to the preset's display setting.
Example:
uv run mainline.py --preset border-test --display websocket
# Now correctly uses websocket instead of terminal (border-test default)
BUG FIXES:
1. Border parameter not being passed to display.show()
- Display backends support border parameter but app.py wasn't passing it
- Now app.py passes params.border to display.show(border=params.border)
- Enables border-test preset to actually render borders
2. WebSocket and Multi displays didn't support border parameter
- Updated WebSocket Protocol to include border parameter
- Updated MultiDisplay.show() to accept and forward border parameter
- Updated test to expect border parameter in mock calls
3. app.py didn't properly handle special sources (empty, pipeline-inspect)
- Border-test preset with source='empty' was still fetching headlines
- Pipeline-inspect source was never using the introspection data source
- Now app.py detects special sources and uses appropriate data source stages:
* 'empty' source → EmptyDataSource stage
* 'pipeline-inspect' → PipelineIntrospectionSource stage
* Other sources → traditional items-based approach
- Uses SourceItemsToBufferStage for special sources instead of RenderStage
- Sets pipeline on introspection source after build to avoid circular dependency
TESTING:
- All 463 tests pass
- Linting passes
- Manual test: `uv run mainline.py --preset border-test` now correctly shows empty source
- border-test preset now properly initializes without fetching unnecessary content
The issue was that the enhanced app.py code from the original diff didn't make it into
the refactor commit. This fix restores that functionality.
- Remove ~700 lines of legacy code from app.py (run_demo_mode, run_pipeline_demo,
run_preset_mode, font picker, effects picker)
- HUD now reads metrics from pipeline context (first-class citizen) with fallback
to global monitor for backwards compatibility
- Add validate_signal_flow() for PureData-style type validation in presets
- Update MicSensor documentation (self-contained, doesn't use MicMonitor)
- Delete test_app.py (was testing removed legacy code)
- Update AGENTS.md with pipeline architecture documentation
- Add beautiful-mermaid library (single-file ASCII renderer)
- Update pipeline_viz to generate mermaid graphs and render with beautiful-mermaid
- Creates dimensional network visualization with arrows connecting nodes
- Animates through effects and highlights active camera mode
- Add --pipeline-diagram flag to generate mermaid diagrams
- Create engine/pipeline.py with PipelineIntrospector
- Outputs flowchart, sequence diagram, and camera state diagram
- Run with: python mainline.py --pipeline-diagram
- Add Camera class with modes: vertical, horizontal, omni, floating
- Refactor scroll.py and demo to use Camera abstraction
- Add vis_offset for horizontal scrolling support
- Add camera_x to EffectContext for effects
- Add pygame window resize handling
- Add HUD effect plugin for demo mode
- Add --demo flag to run demo mode
- Add tests for Camera and vis_offset
- Add HUD effect plugin showing FPS, effect name, intensity bar, pipeline
- Add pygame window resize handling (VIDEORESIZE event)
- Move HUD to end of chain so it renders on top
- Fix monitor stats API (returns dict, not object)
- Add tests/test_hud.py for HUD effect verification
- Fetch real news/poetry content instead of random letters
- Render full ticker zone with scroll, gradients, firehose
- Demo now shows actual effect behavior on real content
- Add --demo flag that runs effect showcase with pygame display
- Add HUD effect plugin (effects_plugins/hud.py) that displays:
- FPS and frame time
- Current effect name with intensity bar
- Pipeline order
- Demo mode cycles through noise, fade, glitch, firehose effects
- Ramps intensity 0→1→0 over 5 seconds per effect
- Fix pre-existing lint errors in engine/ modules using ruff --unsafe-fixes
- Add hk.pkl with pre-commit and pre-push hooks using ruff builtin
- Configure hooks to use 'uv run' prefix for tool execution
- Update mise.toml to include hk and pkl tools
- All 73 tests pass
fix: apply ruff auto-fixes and add hk git hooks
- Fix pre-existing lint errors in engine/ modules using ruff --unsafe-fixes
- Add hk.pkl with pre-commit and pre-push hooks using ruff builtin
- Configure hooks to use 'uv run' prefix for tool execution
- Update mise.toml to include hk and pkl tools
- Use 'hk install --mise' for proper mise integration
- All 73 tests pass
- Fix import sorting (isort) across all engine modules
- Fix SIM105 try-except-pass patterns (contextlib.suppress)
- Fix nested with statements in tests
- Fix unused loop variables
Run 'uv run pytest' to verify tests still pass.
