## Summary
Fixed critical performance issue where demo/poetry presets would hang for 10+ seconds due to FontStage rendering all 1438+ headline items instead of just the visible ~5 items.
## Changes
### Core Fix: ViewportFilterStage
- New pipeline stage that filters items to only those fitting in the viewport
- Reduces 1438 items → ~5 items (288x reduction) before FontStage
- Prevents expensive PIL font rendering operations on items that won't be displayed
- Located: engine/pipeline/adapters.py:348-403
### Pipeline Integration
- Updated app.py to add ViewportFilterStage before FontStage for headlines/poetry sources
- Ensures correct data flow: source → viewport_filter → font → camera → effects → display
- ViewportFilterStage depends on 'source' capability, providing pass-through filtering
### Display Protocol Enhancement
- Added is_quit_requested() and clear_quit_request() method signatures to Display protocol
- Documented as optional methods for backends supporting keyboard input
- Already implemented by pygame backend, now formally part of protocol
### Debug Infrastructure
- Added MAINLINE_DEBUG_DATAFLOW environment variable logging throughout pipeline
- Logs stage input/output types and data sizes to stderr (when flag enabled)
- Verified working: 1438 → 5 item reduction shown in debug output
### Performance Testing
- Added pytest-benchmark (v5.2.3) as dev dependency for statistical benchmarking
- Created comprehensive performance regression tests (tests/test_performance_regression.py)
- Tests verify:
- ViewportFilterStage filters 2000 items efficiently (<1ms)
- FontStage processes filtered items quickly (<50ms)
- 288x performance improvement ratio maintained
- Pipeline doesn't hang with large datasets
- All 523 tests passing, including 7 new performance tests
## Performance Impact
**Before:** FontStage renders all 1438 items per frame → 10+ second hang
**After:** FontStage renders ~5 items per frame → sub-second execution
Real-world impact: Demo preset now responsive and usable with news sources.
## Testing
- Unit tests: 523 passed, 16 skipped
- Regression tests: Catch performance degradation with large datasets
- E2E verification: Debug logging confirms correct pipeline flow
- Benchmark suite: Statistical performance tracking enabled
Critical fix for display rendering:
1. DisplayStage Missing Dependency
- DisplayStage had empty dependencies, causing it to execute before render
- No data was reaching the display output
- Fix: Add 'render.output' dependency so display comes after render stages
- Now proper execution order: source → render → display
2. create_default_pipeline Missing Render Stage
- Default pipeline only had source and display, no render stage between
- Would fail validation with 'Missing render.output' capability
- Fix: Add SourceItemsToBufferStage to convert items to text buffer
- Now complete data flow: source → render → display
3. Updated Test Expectations
- test_display_stage_dependencies now expects 'render.output' dependency
Result: Display backends (pygame, terminal, websocket) now receive proper
rendered text buffers and can display output correctly.
Tests: All 502 tests passing
MAJOR REFACTORING: Consolidate duplicated pipeline code and standardize on
capability-based dependency resolution. This is a significant but backwards-compatible
restructuring that improves maintainability and extensibility.
## ARCHITECTURE CHANGES
### Data Sources Consolidation
- Move engine/sources_v2.py → engine/data_sources/sources.py
- Move engine/pipeline_sources/ → engine/data_sources/
- Create unified DataSource ABC with common interface:
* fetch() - idempotent data retrieval
* get_items() - cached access with automatic refresh
* refresh() - force cache invalidation
* is_dynamic - indicate streaming vs static sources
- Support for SourceItem dataclass (content, source, timestamp, metadata)
### Display Backend Improvements
- Update all 7 display backends to use new import paths
- Terminal: Improve dimension detection and handling
- WebSocket: Better error handling and client lifecycle
- Sixel: Refactor graphics rendering
- Pygame: Modernize event handling
- Kitty: Add protocol support for inline images
- Multi: Ensure proper forwarding to all backends
- Null: Maintain testing backend functionality
### Pipeline Adapter Consolidation
- Refactor adapter stages for clarity and flexibility
- RenderStage now handles both item-based and buffer-based rendering
- Add SourceItemsToBufferStage for converting data source items
- Improve DataSourceStage to work with all source types
- Add DisplayStage wrapper for display backends
### Camera & Viewport Refinements
- Update Camera class for new architecture
- Improve viewport dimension detection
- Better handling of resize events across backends
### New Effect Plugins
- border.py: Frame rendering effect with configurable style
- crop.py: Viewport clipping effect for selective display
- tint.py: Color filtering effect for atmosphere
### Tests & Quality
- Add test_border_effect.py with comprehensive border tests
- Add test_crop_effect.py with viewport clipping tests
- Add test_tint_effect.py with color filtering tests
- Update test_pipeline.py for new architecture
- Update test_pipeline_introspection.py for new data source location
- All 463 tests pass with 56% coverage
- Linting: All checks pass with ruff
### Removals (Code Cleanup)
- Delete engine/benchmark.py (deprecated performance testing)
- Delete engine/pipeline_sources/__init__.py (moved to data_sources)
- Delete engine/sources_v2.py (replaced by data_sources/sources.py)
- Update AGENTS.md to reflect new structure
### Import Path Updates
- Update engine/pipeline/controller.py::create_default_pipeline()
* Old: from engine.sources_v2 import HeadlinesDataSource
* New: from engine.data_sources.sources import HeadlinesDataSource
- All display backends import from new locations
- All tests import from new locations
## BACKWARDS COMPATIBILITY
This refactoring is intended to be backwards compatible:
- Pipeline execution unchanged (DAG-based with capability matching)
- Effect plugins unchanged (EffectPlugin interface same)
- Display protocol unchanged (Display duck-typing works as before)
- Config system unchanged (presets.toml format same)
## TESTING
- 463 tests pass (0 failures, 19 skipped)
- Full linting check passes
- Manual testing on demo, poetry, websocket modes
- All new effect plugins tested
## FILES CHANGED
- 24 files modified/added/deleted
- 723 insertions, 1,461 deletions (net -738 LOC - cleanup!)
- No breaking changes to public APIs
- All transitive imports updated correctly
- Add PipelineIntrospectionSource that renders live ASCII DAG with metrics
- Add PipelineMetricsSensor exposing pipeline performance as sensor values
- Add PipelineIntrospectionDemo controller with 3-phase animation:
- Phase 1: Toggle effects one at a time (3s each)
- Phase 2: LFO drives intensity default→max→min→default
- Phase 3: All effects with shared LFO (infinite loop)
- Add pipeline-inspect preset
- Add get_frame_times() to Pipeline for sparkline data
- Add tests for new components
- Update mise.toml with pipeline-inspect preset task
- Add DataType enum (SOURCE_ITEMS, TEXT_BUFFER, etc.)
- Add inlet_types and outlet_types to Stage
- Add _validate_types() for type checking at build time
- Update tests with proper type annotations
- Add TransformDataSource for filtering/mapping source items
- Add MetricsDataSource for rendering live pipeline metrics as ASCII art
- Fix display stage registration in StageRegistry
- Register sources with both class name and simple name aliases
- Fix DisplayStage.init() to pass reuse parameter
- Simplify create_default_pipeline to use DataSourceStage wrapper
- Set pygame as default display
- Remove old pipeline tasks from mise.toml
- Add tests for new pipeline architecture
