refactor: consolidate pipeline architecture with unified data source system
MAJOR REFACTORING: Consolidate duplicated pipeline code and standardize on capability-based dependency resolution. This is a significant but backwards-compatible restructuring that improves maintainability and extensibility. ## ARCHITECTURE CHANGES ### Data Sources Consolidation - Move engine/sources_v2.py → engine/data_sources/sources.py - Move engine/pipeline_sources/ → engine/data_sources/ - Create unified DataSource ABC with common interface: * fetch() - idempotent data retrieval * get_items() - cached access with automatic refresh * refresh() - force cache invalidation * is_dynamic - indicate streaming vs static sources - Support for SourceItem dataclass (content, source, timestamp, metadata) ### Display Backend Improvements - Update all 7 display backends to use new import paths - Terminal: Improve dimension detection and handling - WebSocket: Better error handling and client lifecycle - Sixel: Refactor graphics rendering - Pygame: Modernize event handling - Kitty: Add protocol support for inline images - Multi: Ensure proper forwarding to all backends - Null: Maintain testing backend functionality ### Pipeline Adapter Consolidation - Refactor adapter stages for clarity and flexibility - RenderStage now handles both item-based and buffer-based rendering - Add SourceItemsToBufferStage for converting data source items - Improve DataSourceStage to work with all source types - Add DisplayStage wrapper for display backends ### Camera & Viewport Refinements - Update Camera class for new architecture - Improve viewport dimension detection - Better handling of resize events across backends ### New Effect Plugins - border.py: Frame rendering effect with configurable style - crop.py: Viewport clipping effect for selective display - tint.py: Color filtering effect for atmosphere ### Tests & Quality - Add test_border_effect.py with comprehensive border tests - Add test_crop_effect.py with viewport clipping tests - Add test_tint_effect.py with color filtering tests - Update test_pipeline.py for new architecture - Update test_pipeline_introspection.py for new data source location - All 463 tests pass with 56% coverage - Linting: All checks pass with ruff ### Removals (Code Cleanup) - Delete engine/benchmark.py (deprecated performance testing) - Delete engine/pipeline_sources/__init__.py (moved to data_sources) - Delete engine/sources_v2.py (replaced by data_sources/sources.py) - Update AGENTS.md to reflect new structure ### Import Path Updates - Update engine/pipeline/controller.py::create_default_pipeline() * Old: from engine.sources_v2 import HeadlinesDataSource * New: from engine.data_sources.sources import HeadlinesDataSource - All display backends import from new locations - All tests import from new locations ## BACKWARDS COMPATIBILITY This refactoring is intended to be backwards compatible: - Pipeline execution unchanged (DAG-based with capability matching) - Effect plugins unchanged (EffectPlugin interface same) - Display protocol unchanged (Display duck-typing works as before) - Config system unchanged (presets.toml format same) ## TESTING - 463 tests pass (0 failures, 19 skipped) - Full linting check passes - Manual testing on demo, poetry, websocket modes - All new effect plugins tested ## FILES CHANGED - 24 files modified/added/deleted - 723 insertions, 1,461 deletions (net -738 LOC - cleanup!) - No breaking changes to public APIs - All transitive imports updated correctly
This commit is contained in:
43
AGENTS.md
43
AGENTS.md
@@ -71,39 +71,17 @@ The project uses hk configured in `hk.pkl`:
|
||||
|
||||
## Benchmark Runner
|
||||
|
||||
Run performance benchmarks:
|
||||
Benchmark tests are in `tests/test_benchmark.py` with `@pytest.mark.benchmark`.
|
||||
|
||||
### Hook Mode (via pytest)
|
||||
|
||||
Run benchmarks in hook mode to catch performance regressions:
|
||||
|
||||
```bash
|
||||
mise run benchmark # Run all benchmarks (text output)
|
||||
mise run benchmark-json # Run benchmarks (JSON output)
|
||||
mise run benchmark-report # Run benchmarks (Markdown report)
|
||||
mise run test-cov # Run with coverage
|
||||
```
|
||||
|
||||
### Benchmark Commands
|
||||
|
||||
```bash
|
||||
# Run benchmarks
|
||||
uv run python -m engine.benchmark
|
||||
|
||||
# Run with specific displays/effects
|
||||
uv run python -m engine.benchmark --displays null,terminal --effects fade,glitch
|
||||
|
||||
# Save baseline for hook comparisons
|
||||
uv run python -m engine.benchmark --baseline
|
||||
|
||||
# Run in hook mode (compares against baseline)
|
||||
uv run python -m engine.benchmark --hook
|
||||
|
||||
# Hook mode with custom threshold (default: 20% degradation)
|
||||
uv run python -m engine.benchmark --hook --threshold 0.3
|
||||
|
||||
# Custom baseline location
|
||||
uv run python -m engine.benchmark --hook --cache /path/to/cache.json
|
||||
```
|
||||
|
||||
### Hook Mode
|
||||
|
||||
The `--hook` mode compares current benchmarks against a saved baseline. If performance degrades beyond the threshold (default 20%), it exits with code 1. This is useful for preventing performance regressions in feature branches.
|
||||
The benchmark tests will fail if performance degrades beyond the threshold.
|
||||
|
||||
The pre-push hook runs benchmark in hook mode to catch performance regressions before pushing.
|
||||
|
||||
@@ -161,12 +139,11 @@ The project uses pytest with strict marker enforcement. Test configuration is in
|
||||
|
||||
### Test Coverage Strategy
|
||||
|
||||
Current coverage: 56% (434 tests)
|
||||
Current coverage: 56% (463 tests)
|
||||
|
||||
Key areas with lower coverage (acceptable for now):
|
||||
- **app.py** (8%): Main entry point - integration heavy, requires terminal
|
||||
- **scroll.py** (10%): Terminal-dependent rendering logic
|
||||
- **benchmark.py** (0%): Standalone benchmark tool, runs separately
|
||||
- **scroll.py** (10%): Terminal-dependent rendering logic (unused)
|
||||
|
||||
Key areas with good coverage:
|
||||
- **display/backends/null.py** (95%): Easy to test headlessly
|
||||
@@ -227,7 +204,7 @@ Sensors support param bindings to drive effect parameters in real-time.
|
||||
|
||||
#### Pipeline Introspection
|
||||
|
||||
- **PipelineIntrospectionSource** (`engine/pipeline_sources/pipeline_introspection.py`): Renders live ASCII visualization of pipeline DAG with metrics
|
||||
- **PipelineIntrospectionSource** (`engine/data_sources/pipeline_introspection.py`): Renders live ASCII visualization of pipeline DAG with metrics
|
||||
- **PipelineIntrospectionDemo** (`engine/pipeline/pipeline_introspection_demo.py`): 3-phase demo controller for effect animation
|
||||
|
||||
Preset: `pipeline-inspect` - Live pipeline introspection with DAG and performance metrics
|
||||
|
||||
Reference in New Issue
Block a user