MAJOR REFACTORING: Consolidate duplicated pipeline code and standardize on
capability-based dependency resolution. This is a significant but backwards-compatible
restructuring that improves maintainability and extensibility.
## ARCHITECTURE CHANGES
### Data Sources Consolidation
- Move engine/sources_v2.py → engine/data_sources/sources.py
- Move engine/pipeline_sources/ → engine/data_sources/
- Create unified DataSource ABC with common interface:
* fetch() - idempotent data retrieval
* get_items() - cached access with automatic refresh
* refresh() - force cache invalidation
* is_dynamic - indicate streaming vs static sources
- Support for SourceItem dataclass (content, source, timestamp, metadata)
### Display Backend Improvements
- Update all 7 display backends to use new import paths
- Terminal: Improve dimension detection and handling
- WebSocket: Better error handling and client lifecycle
- Sixel: Refactor graphics rendering
- Pygame: Modernize event handling
- Kitty: Add protocol support for inline images
- Multi: Ensure proper forwarding to all backends
- Null: Maintain testing backend functionality
### Pipeline Adapter Consolidation
- Refactor adapter stages for clarity and flexibility
- RenderStage now handles both item-based and buffer-based rendering
- Add SourceItemsToBufferStage for converting data source items
- Improve DataSourceStage to work with all source types
- Add DisplayStage wrapper for display backends
### Camera & Viewport Refinements
- Update Camera class for new architecture
- Improve viewport dimension detection
- Better handling of resize events across backends
### New Effect Plugins
- border.py: Frame rendering effect with configurable style
- crop.py: Viewport clipping effect for selective display
- tint.py: Color filtering effect for atmosphere
### Tests & Quality
- Add test_border_effect.py with comprehensive border tests
- Add test_crop_effect.py with viewport clipping tests
- Add test_tint_effect.py with color filtering tests
- Update test_pipeline.py for new architecture
- Update test_pipeline_introspection.py for new data source location
- All 463 tests pass with 56% coverage
- Linting: All checks pass with ruff
### Removals (Code Cleanup)
- Delete engine/benchmark.py (deprecated performance testing)
- Delete engine/pipeline_sources/__init__.py (moved to data_sources)
- Delete engine/sources_v2.py (replaced by data_sources/sources.py)
- Update AGENTS.md to reflect new structure
### Import Path Updates
- Update engine/pipeline/controller.py::create_default_pipeline()
* Old: from engine.sources_v2 import HeadlinesDataSource
* New: from engine.data_sources.sources import HeadlinesDataSource
- All display backends import from new locations
- All tests import from new locations
## BACKWARDS COMPATIBILITY
This refactoring is intended to be backwards compatible:
- Pipeline execution unchanged (DAG-based with capability matching)
- Effect plugins unchanged (EffectPlugin interface same)
- Display protocol unchanged (Display duck-typing works as before)
- Config system unchanged (presets.toml format same)
## TESTING
- 463 tests pass (0 failures, 19 skipped)
- Full linting check passes
- Manual testing on demo, poetry, websocket modes
- All new effect plugins tested
## FILES CHANGED
- 24 files modified/added/deleted
- 723 insertions, 1,461 deletions (net -738 LOC - cleanup!)
- No breaking changes to public APIs
- All transitive imports updated correctly
- Add PartialUpdate dataclass and supports_partial_updates to EffectPlugin
- Add dirty region tracking to Canvas (mark_dirty, get_dirty_rows, etc.)
- Canvas auto-marks dirty on put_region, put_text, fill
- CanvasStage exposes dirty rows via pipeline context
- EffectChain creates PartialUpdate and calls process_partial() for optimized effects
- HudEffect implements process_partial() to skip processing when rows 0-2 not dirty
- This enables effects to skip work when canvas regions haven't changed
- Remove ~700 lines of legacy code from app.py (run_demo_mode, run_pipeline_demo,
run_preset_mode, font picker, effects picker)
- HUD now reads metrics from pipeline context (first-class citizen) with fallback
to global monitor for backwards compatibility
- Add validate_signal_flow() for PureData-style type validation in presets
- Update MicSensor documentation (self-contained, doesn't use MicMonitor)
- Delete test_app.py (was testing removed legacy code)
- Update AGENTS.md with pipeline architecture documentation
- Add HUD effect plugin showing FPS, effect name, intensity bar, pipeline
- Add pygame window resize handling (VIDEORESIZE event)
- Move HUD to end of chain so it renders on top
- Fix monitor stats API (returns dict, not object)
- Add tests/test_hud.py for HUD effect verification
- Add --demo flag that runs effect showcase with pygame display
- Add HUD effect plugin (effects_plugins/hud.py) that displays:
- FPS and frame time
- Current effect name with intensity bar
- Pipeline order
- Demo mode cycles through noise, fade, glitch, firehose effects
- Ramps intensity 0→1→0 over 5 seconds per effect
