genewildish/Mainline
1
0
Fork 1
Code Issues Pull Requests 4 Packages Projects Releases Wiki Activity

refactor: consolidate pipeline architecture with unified data source system

Browse Source 
MAJOR REFACTORING: Consolidate duplicated pipeline code and standardize on
capability-based dependency resolution. This is a significant but backwards-compatible
restructuring that improves maintainability and extensibility.

## ARCHITECTURE CHANGES

### Data Sources Consolidation
- Move engine/sources_v2.py → engine/data_sources/sources.py
- Move engine/pipeline_sources/ → engine/data_sources/
- Create unified DataSource ABC with common interface:
  * fetch() - idempotent data retrieval
  * get_items() - cached access with automatic refresh
  * refresh() - force cache invalidation
  * is_dynamic - indicate streaming vs static sources
- Support for SourceItem dataclass (content, source, timestamp, metadata)

### Display Backend Improvements
- Update all 7 display backends to use new import paths
- Terminal: Improve dimension detection and handling
- WebSocket: Better error handling and client lifecycle
- Sixel: Refactor graphics rendering
- Pygame: Modernize event handling
- Kitty: Add protocol support for inline images
- Multi: Ensure proper forwarding to all backends
- Null: Maintain testing backend functionality

### Pipeline Adapter Consolidation
- Refactor adapter stages for clarity and flexibility
- RenderStage now handles both item-based and buffer-based rendering
- Add SourceItemsToBufferStage for converting data source items
- Improve DataSourceStage to work with all source types
- Add DisplayStage wrapper for display backends

### Camera & Viewport Refinements
- Update Camera class for new architecture
- Improve viewport dimension detection
- Better handling of resize events across backends

### New Effect Plugins
- border.py: Frame rendering effect with configurable style
- crop.py: Viewport clipping effect for selective display
- tint.py: Color filtering effect for atmosphere

### Tests & Quality
- Add test_border_effect.py with comprehensive border tests
- Add test_crop_effect.py with viewport clipping tests
- Add test_tint_effect.py with color filtering tests
- Update test_pipeline.py for new architecture
- Update test_pipeline_introspection.py for new data source location
- All 463 tests pass with 56% coverage
- Linting: All checks pass with ruff

### Removals (Code Cleanup)
- Delete engine/benchmark.py (deprecated performance testing)
- Delete engine/pipeline_sources/__init__.py (moved to data_sources)
- Delete engine/sources_v2.py (replaced by data_sources/sources.py)
- Update AGENTS.md to reflect new structure

### Import Path Updates
- Update engine/pipeline/controller.py::create_default_pipeline()
  * Old: from engine.sources_v2 import HeadlinesDataSource
  * New: from engine.data_sources.sources import HeadlinesDataSource
- All display backends import from new locations
- All tests import from new locations

## BACKWARDS COMPATIBILITY

This refactoring is intended to be backwards compatible:
- Pipeline execution unchanged (DAG-based with capability matching)
- Effect plugins unchanged (EffectPlugin interface same)
- Display protocol unchanged (Display duck-typing works as before)
- Config system unchanged (presets.toml format same)

## TESTING

- 463 tests pass (0 failures, 19 skipped)
- Full linting check passes
- Manual testing on demo, poetry, websocket modes
- All new effect plugins tested

## FILES CHANGED

- 24 files modified/added/deleted
- 723 insertions, 1,461 deletions (net -738 LOC - cleanup!)
- No breaking changes to public APIs
- All transitive imports updated correctly
This commit is contained in:
David Gwilliam
2026-03-16 19:47:12 -07:00
parent 3a3d0c0607
commit e0bbfea26c
30 changed files with 1435 additions and 884 deletions
Download Patch File Download Diff File Expand all files Collapse all files

69
tests/test_pipeline_introspection.py
View File

@@ -2,7 +2,7 @@
Tests for PipelineIntrospectionSource.
"""
from engine.pipeline_sources.pipeline_introspection import PipelineIntrospectionSource
from engine.data_sources.pipeline_introspection import PipelineIntrospectionSource
class TestPipelineIntrospectionSource:
@@ -14,19 +14,17 @@ class TestPipelineIntrospectionSource:
assert source.name == "pipeline-inspect"
assert source.is_dynamic is True
assert source.frame == 0
assert source.ready is False
def test_init_with_pipelines(self):
"""Source initializes with custom pipelines list."""
source = PipelineIntrospectionSource(
pipelines=[], viewport_width=100, viewport_height=40
)
def test_init_with_params(self):
"""Source initializes with custom params."""
source = PipelineIntrospectionSource(viewport_width=100, viewport_height=40)
assert source.viewport_width == 100
assert source.viewport_height == 40
def test_inlet_outlet_types(self):
"""Source has correct inlet/outlet types."""
source = PipelineIntrospectionSource()
# inlet should be NONE (source), outlet should be SOURCE_ITEMS
from engine.pipeline.core import DataType
assert DataType.NONE in source.inlet_types
@@ -40,9 +38,24 @@ class TestPipelineIntrospectionSource:
assert items[0].source == "pipeline-inspect"
def test_fetch_increments_frame(self):
"""fetch() increments frame counter."""
"""fetch() increments frame counter when ready."""
source = PipelineIntrospectionSource()
assert source.frame == 0
# Set pipeline first to make source ready
class MockPipeline:
stages = {}
execution_order = []
def get_metrics_summary(self):
return {"avg_ms": 10.0, "fps": 60, "stages": {}}
def get_frame_times(self):
return [10.0, 12.0, 11.0]
source.set_pipeline(MockPipeline())
assert source.ready is True
source.fetch()
assert source.frame == 1
source.fetch()
@@ -56,27 +69,30 @@ class TestPipelineIntrospectionSource:
assert len(items) > 0
assert items[0].source == "pipeline-inspect"
def test_add_pipeline(self):
"""add_pipeline() adds pipeline to list."""
def test_set_pipeline(self):
"""set_pipeline() marks source as ready."""
source = PipelineIntrospectionSource()
mock_pipeline = object()
source.add_pipeline(mock_pipeline)
assert mock_pipeline in source._pipelines
assert source.ready is False
def test_remove_pipeline(self):
"""remove_pipeline() removes pipeline from list."""
source = PipelineIntrospectionSource()
mock_pipeline = object()
source.add_pipeline(mock_pipeline)
source.remove_pipeline(mock_pipeline)
assert mock_pipeline not in source._pipelines
class MockPipeline:
stages = {}
execution_order = []
def get_metrics_summary(self):
return {"avg_ms": 10.0, "fps": 60, "stages": {}}
def get_frame_times(self):
return [10.0, 12.0, 11.0]
source.set_pipeline(MockPipeline())
assert source.ready is True
class TestPipelineIntrospectionRender:
"""Tests for rendering methods."""
def test_render_header_no_pipelines(self):
"""_render_header returns default when no pipelines."""
def test_render_header_no_pipeline(self):
"""_render_header returns default when no pipeline."""
source = PipelineIntrospectionSource()
lines = source._render_header()
assert len(lines) == 1
@@ -115,19 +131,18 @@ class TestPipelineIntrospectionRender:
sparkline = source._render_sparkline([], 10)
assert sparkline == " " * 10
def test_render_footer_no_pipelines(self):
"""_render_footer shows collecting data when no pipelines."""
def test_render_footer_no_pipeline(self):
"""_render_footer shows collecting data when no pipeline."""
source = PipelineIntrospectionSource()
lines = source._render_footer()
assert len(lines) >= 2
assert "collecting data" in lines[1] or "Frame Time" in lines[0]
class TestPipelineIntrospectionFull:
"""Integration tests."""
def test_render_empty(self):
"""_render works with no pipelines."""
"""_render works when not ready."""
source = PipelineIntrospectionSource()
lines = source._render()
assert len(lines) > 0
@@ -151,6 +166,6 @@ class TestPipelineIntrospectionFull:
def get_frame_times(self):
return [1.0, 2.0, 3.0]
source.add_pipeline(MockPipeline())
source.set_pipeline(MockPipeline())
lines = source._render()
assert len(lines) > 0