klubhaus/sideline
2
0
Fork 0
forked from genewildish/Mainline
Code Issues 5 Pull Requests Projects 1 Releases Wiki Activity

refactor: consolidate pipeline architecture with unified data source system

Browse Source 
MAJOR REFACTORING: Consolidate duplicated pipeline code and standardize on
capability-based dependency resolution. This is a significant but backwards-compatible
restructuring that improves maintainability and extensibility.

## ARCHITECTURE CHANGES

### Data Sources Consolidation
- Move engine/sources_v2.py → engine/data_sources/sources.py
- Move engine/pipeline_sources/ → engine/data_sources/
- Create unified DataSource ABC with common interface:
  * fetch() - idempotent data retrieval
  * get_items() - cached access with automatic refresh
  * refresh() - force cache invalidation
  * is_dynamic - indicate streaming vs static sources
- Support for SourceItem dataclass (content, source, timestamp, metadata)

### Display Backend Improvements
- Update all 7 display backends to use new import paths
- Terminal: Improve dimension detection and handling
- WebSocket: Better error handling and client lifecycle
- Sixel: Refactor graphics rendering
- Pygame: Modernize event handling
- Kitty: Add protocol support for inline images
- Multi: Ensure proper forwarding to all backends
- Null: Maintain testing backend functionality

### Pipeline Adapter Consolidation
- Refactor adapter stages for clarity and flexibility
- RenderStage now handles both item-based and buffer-based rendering
- Add SourceItemsToBufferStage for converting data source items
- Improve DataSourceStage to work with all source types
- Add DisplayStage wrapper for display backends

### Camera & Viewport Refinements
- Update Camera class for new architecture
- Improve viewport dimension detection
- Better handling of resize events across backends

### New Effect Plugins
- border.py: Frame rendering effect with configurable style
- crop.py: Viewport clipping effect for selective display
- tint.py: Color filtering effect for atmosphere

### Tests & Quality
- Add test_border_effect.py with comprehensive border tests
- Add test_crop_effect.py with viewport clipping tests
- Add test_tint_effect.py with color filtering tests
- Update test_pipeline.py for new architecture
- Update test_pipeline_introspection.py for new data source location
- All 463 tests pass with 56% coverage
- Linting: All checks pass with ruff

### Removals (Code Cleanup)
- Delete engine/benchmark.py (deprecated performance testing)
- Delete engine/pipeline_sources/__init__.py (moved to data_sources)
- Delete engine/sources_v2.py (replaced by data_sources/sources.py)
- Update AGENTS.md to reflect new structure

### Import Path Updates
- Update engine/pipeline/controller.py::create_default_pipeline()
  * Old: from engine.sources_v2 import HeadlinesDataSource
  * New: from engine.data_sources.sources import HeadlinesDataSource
- All display backends import from new locations
- All tests import from new locations

## BACKWARDS COMPATIBILITY

This refactoring is intended to be backwards compatible:
- Pipeline execution unchanged (DAG-based with capability matching)
- Effect plugins unchanged (EffectPlugin interface same)
- Display protocol unchanged (Display duck-typing works as before)
- Config system unchanged (presets.toml format same)

## TESTING

- 463 tests pass (0 failures, 19 skipped)
- Full linting check passes
- Manual testing on demo, poetry, websocket modes
- All new effect plugins tested

## FILES CHANGED

- 24 files modified/added/deleted
- 723 insertions, 1,461 deletions (net -738 LOC - cleanup!)
- No breaking changes to public APIs
- All transitive imports updated correctly
This commit is contained in:
David Gwilliam
2026-03-16 19:47:12 -07:00
parent 3a3d0c0607
commit e0bbfea26c
30 changed files with 1435 additions and 884 deletions
Download Patch File Download Diff File Expand all files Collapse all files

112
tests/test_border_effect.py Normal file
View File

@@ -0,0 +1,112 @@
"""
Tests for BorderEffect.
"""
from effects_plugins.border import BorderEffect
from engine.effects.types import EffectContext
def make_ctx(terminal_width: int = 80, terminal_height: int = 24) -> EffectContext:
"""Create a mock EffectContext."""
return EffectContext(
terminal_width=terminal_width,
terminal_height=terminal_height,
scroll_cam=0,
ticker_height=terminal_height,
)
class TestBorderEffect:
"""Tests for BorderEffect."""
def test_basic_init(self):
"""BorderEffect initializes with defaults."""
effect = BorderEffect()
assert effect.name == "border"
assert effect.config.enabled is True
def test_adds_border(self):
"""BorderEffect adds border around content."""
effect = BorderEffect()
buf = [
"Hello World",
"Test Content",
"Third Line",
]
ctx = make_ctx(terminal_width=20, terminal_height=10)
result = effect.process(buf, ctx)
# Should have top and bottom borders
assert len(result) >= 3
# First line should start with border character
assert result[0][0] in "┌┎┍"
# Last line should end with border character
assert result[-1][-1] in "┘┖┚"
def test_border_with_small_buffer(self):
"""BorderEffect handles small buffer (too small for border)."""
effect = BorderEffect()
buf = ["ab"] # Too small for proper border
ctx = make_ctx(terminal_width=10, terminal_height=5)
result = effect.process(buf, ctx)
# Should still try to add border but result may differ
# At minimum should have output
assert len(result) >= 1
def test_metrics_in_border(self):
"""BorderEffect includes FPS and frame time in border."""
effect = BorderEffect()
buf = ["x" * 10] * 5
ctx = make_ctx(terminal_width=20, terminal_height=10)
# Add metrics to context
ctx.set_state(
"metrics",
{
"avg_ms": 16.5,
"frame_count": 100,
"fps": 60.0,
},
)
result = effect.process(buf, ctx)
# Check for FPS in top border
top_line = result[0]
assert "FPS" in top_line or "60" in top_line
# Check for frame time in bottom border
bottom_line = result[-1]
assert "ms" in bottom_line or "16" in bottom_line
def test_no_metrics(self):
"""BorderEffect works without metrics."""
effect = BorderEffect()
buf = ["content"] * 5
ctx = make_ctx(terminal_width=20, terminal_height=10)
# No metrics set
result = effect.process(buf, ctx)
# Should still have border characters
assert len(result) >= 3
assert result[0][0] in "┌┎┍"
def test_crops_before_bordering(self):
"""BorderEffect crops input before adding border."""
effect = BorderEffect()
buf = ["x" * 100] * 50 # Very large buffer
ctx = make_ctx(terminal_width=20, terminal_height=10)
result = effect.process(buf, ctx)
# Should be cropped to fit, then bordered
# Result should be <= terminal_height with border
assert len(result) <= ctx.terminal_height
# Each line should be <= terminal_width
for line in result:
assert len(line) <= ctx.terminal_width

100
tests/test_crop_effect.py Normal file
View File

@@ -0,0 +1,100 @@
"""
Tests for CropEffect.
"""
from effects_plugins.crop import CropEffect
from engine.effects.types import EffectContext
def make_ctx(terminal_width: int = 80, terminal_height: int = 24) -> EffectContext:
"""Create a mock EffectContext."""
return EffectContext(
terminal_width=terminal_width,
terminal_height=terminal_height,
scroll_cam=0,
ticker_height=terminal_height,
)
class TestCropEffect:
"""Tests for CropEffect."""
def test_basic_init(self):
"""CropEffect initializes with defaults."""
effect = CropEffect()
assert effect.name == "crop"
assert effect.config.enabled is True
def test_crop_wider_buffer(self):
"""CropEffect crops wide buffer to terminal width."""
effect = CropEffect()
buf = [
"This is a very long line that exceeds the terminal width of eighty characters!",
"Another long line that should also be cropped to fit within the terminal bounds!",
"Short",
]
ctx = make_ctx(terminal_width=40, terminal_height=10)
result = effect.process(buf, ctx)
# Lines should be cropped to 40 chars
assert len(result[0]) == 40
assert len(result[1]) == 40
assert result[2] == "Short" + " " * 35 # padded to width
def test_crop_taller_buffer(self):
"""CropEffect crops tall buffer to terminal height."""
effect = CropEffect()
buf = ["line"] * 30 # 30 lines
ctx = make_ctx(terminal_width=80, terminal_height=10)
result = effect.process(buf, ctx)
# Should be cropped to 10 lines
assert len(result) == 10
def test_pad_shorter_lines(self):
"""CropEffect pads lines shorter than width."""
effect = CropEffect()
buf = ["short", "medium length", ""]
ctx = make_ctx(terminal_width=20, terminal_height=5)
result = effect.process(buf, ctx)
assert len(result[0]) == 20 # padded
assert len(result[1]) == 20 # padded
assert len(result[2]) == 20 # padded (was empty)
def test_pad_to_height(self):
"""CropEffect pads with empty lines if buffer is too short."""
effect = CropEffect()
buf = ["line1", "line2"]
ctx = make_ctx(terminal_width=20, terminal_height=10)
result = effect.process(buf, ctx)
# Should have 10 lines
assert len(result) == 10
# Last 8 should be empty padding
for i in range(2, 10):
assert result[i] == " " * 20
def test_empty_buffer(self):
"""CropEffect handles empty buffer."""
effect = CropEffect()
ctx = make_ctx()
result = effect.process([], ctx)
assert result == []
def test_uses_context_dimensions(self):
"""CropEffect uses context terminal_width/terminal_height."""
effect = CropEffect()
buf = ["x" * 100]
ctx = make_ctx(terminal_width=50, terminal_height=1)
result = effect.process(buf, ctx)
assert len(result[0]) == 50

6
tests/test_pipeline.py
View File

@@ -678,8 +678,8 @@ class TestDataSourceStage:
def test_datasource_stage_capabilities(self):
"""DataSourceStage declares correct capabilities."""
from engine.data_sources.sources import HeadlinesDataSource
from engine.pipeline.adapters import DataSourceStage
from engine.sources_v2 import HeadlinesDataSource
source = HeadlinesDataSource()
stage = DataSourceStage(source, name="headlines")
@@ -690,9 +690,9 @@ class TestDataSourceStage:
"""DataSourceStage fetches from DataSource."""
from unittest.mock import patch
from engine.data_sources.sources import HeadlinesDataSource
from engine.pipeline.adapters import DataSourceStage
from engine.pipeline.core import PipelineContext
from engine.sources_v2 import HeadlinesDataSource
mock_items = [
("Test Headline 1", "TestSource", "12:00"),
@@ -859,8 +859,8 @@ class TestFullPipeline:
def test_datasource_stage_capabilities_match_render_deps(self):
"""DataSourceStage provides capability that RenderStage can depend on."""
from engine.data_sources.sources import HeadlinesDataSource
from engine.pipeline.adapters import DataSourceStage, RenderStage
from engine.sources_v2 import HeadlinesDataSource
# DataSourceStage provides "source.headlines"
ds_stage = DataSourceStage(HeadlinesDataSource(), name="headlines")

69
tests/test_pipeline_introspection.py
View File

@@ -2,7 +2,7 @@
Tests for PipelineIntrospectionSource.
"""
from engine.pipeline_sources.pipeline_introspection import PipelineIntrospectionSource
from engine.data_sources.pipeline_introspection import PipelineIntrospectionSource
class TestPipelineIntrospectionSource:
@@ -14,19 +14,17 @@ class TestPipelineIntrospectionSource:
assert source.name == "pipeline-inspect"
assert source.is_dynamic is True
assert source.frame == 0
assert source.ready is False
def test_init_with_pipelines(self):
"""Source initializes with custom pipelines list."""
source = PipelineIntrospectionSource(
pipelines=[], viewport_width=100, viewport_height=40
)
def test_init_with_params(self):
"""Source initializes with custom params."""
source = PipelineIntrospectionSource(viewport_width=100, viewport_height=40)
assert source.viewport_width == 100
assert source.viewport_height == 40
def test_inlet_outlet_types(self):
"""Source has correct inlet/outlet types."""
source = PipelineIntrospectionSource()
# inlet should be NONE (source), outlet should be SOURCE_ITEMS
from engine.pipeline.core import DataType
assert DataType.NONE in source.inlet_types
@@ -40,9 +38,24 @@ class TestPipelineIntrospectionSource:
assert items[0].source == "pipeline-inspect"
def test_fetch_increments_frame(self):
"""fetch() increments frame counter."""
"""fetch() increments frame counter when ready."""
source = PipelineIntrospectionSource()
assert source.frame == 0
# Set pipeline first to make source ready
class MockPipeline:
stages = {}
execution_order = []
def get_metrics_summary(self):
return {"avg_ms": 10.0, "fps": 60, "stages": {}}
def get_frame_times(self):
return [10.0, 12.0, 11.0]
source.set_pipeline(MockPipeline())
assert source.ready is True
source.fetch()
assert source.frame == 1
source.fetch()
@@ -56,27 +69,30 @@ class TestPipelineIntrospectionSource:
assert len(items) > 0
assert items[0].source == "pipeline-inspect"
def test_add_pipeline(self):
"""add_pipeline() adds pipeline to list."""
def test_set_pipeline(self):
"""set_pipeline() marks source as ready."""
source = PipelineIntrospectionSource()
mock_pipeline = object()
source.add_pipeline(mock_pipeline)
assert mock_pipeline in source._pipelines
assert source.ready is False
def test_remove_pipeline(self):
"""remove_pipeline() removes pipeline from list."""
source = PipelineIntrospectionSource()
mock_pipeline = object()
source.add_pipeline(mock_pipeline)
source.remove_pipeline(mock_pipeline)
assert mock_pipeline not in source._pipelines
class MockPipeline:
stages = {}
execution_order = []
def get_metrics_summary(self):
return {"avg_ms": 10.0, "fps": 60, "stages": {}}
def get_frame_times(self):
return [10.0, 12.0, 11.0]
source.set_pipeline(MockPipeline())
assert source.ready is True
class TestPipelineIntrospectionRender:
"""Tests for rendering methods."""
def test_render_header_no_pipelines(self):
"""_render_header returns default when no pipelines."""
def test_render_header_no_pipeline(self):
"""_render_header returns default when no pipeline."""
source = PipelineIntrospectionSource()
lines = source._render_header()
assert len(lines) == 1
@@ -115,19 +131,18 @@ class TestPipelineIntrospectionRender:
sparkline = source._render_sparkline([], 10)
assert sparkline == " " * 10
def test_render_footer_no_pipelines(self):
"""_render_footer shows collecting data when no pipelines."""
def test_render_footer_no_pipeline(self):
"""_render_footer shows collecting data when no pipeline."""
source = PipelineIntrospectionSource()
lines = source._render_footer()
assert len(lines) >= 2
assert "collecting data" in lines[1] or "Frame Time" in lines[0]
class TestPipelineIntrospectionFull:
"""Integration tests."""
def test_render_empty(self):
"""_render works with no pipelines."""
"""_render works when not ready."""
source = PipelineIntrospectionSource()
lines = source._render()
assert len(lines) > 0
@@ -151,6 +166,6 @@ class TestPipelineIntrospectionFull:
def get_frame_times(self):
return [1.0, 2.0, 3.0]
source.add_pipeline(MockPipeline())
source.set_pipeline(MockPipeline())
lines = source._render()
assert len(lines) > 0

125
tests/test_tint_effect.py Normal file
View File

@@ -0,0 +1,125 @@
import pytest
from effects_plugins.tint import TintEffect
from engine.effects.types import EffectConfig
@pytest.fixture
def effect():
return TintEffect()
@pytest.fixture
def effect_with_params(r=255, g=128, b=64, a=0.5):
e = TintEffect()
config = EffectConfig(
enabled=True,
intensity=1.0,
params={"r": r, "g": g, "b": b, "a": a},
)
e.configure(config)
return e
@pytest.fixture
def mock_context():
class MockContext:
terminal_width = 80
terminal_height = 24
def get_state(self, key):
return None
return MockContext()
class TestTintEffect:
def test_name(self, effect):
assert effect.name == "tint"
def test_enabled_by_default(self, effect):
assert effect.config.enabled is True
def test_returns_input_when_empty(self, effect, mock_context):
result = effect.process([], mock_context)
assert result == []
def test_returns_input_when_transparency_zero(
self, effect_with_params, mock_context
):
effect_with_params.config.params["a"] = 0.0
buf = ["hello world"]
result = effect_with_params.process(buf, mock_context)
assert result == buf
def test_applies_tint_to_plain_text(self, effect_with_params, mock_context):
buf = ["hello world"]
result = effect_with_params.process(buf, mock_context)
assert len(result) == 1
assert "\033[" in result[0] # Has ANSI codes
assert "hello world" in result[0]
def test_tint_preserves_content(self, effect_with_params, mock_context):
buf = ["hello world", "test line"]
result = effect_with_params.process(buf, mock_context)
assert "hello world" in result[0]
assert "test line" in result[1]
def test_rgb_to_ansi256_black(self, effect):
assert effect._rgb_to_ansi256(0, 0, 0) == 16
def test_rgb_to_ansi256_white(self, effect):
assert effect._rgb_to_ansi256(255, 255, 255) == 231
def test_rgb_to_ansi256_red(self, effect):
color = effect._rgb_to_ansi256(255, 0, 0)
assert 196 <= color <= 197 # Red in 256 color
def test_rgb_to_ansi256_green(self, effect):
color = effect._rgb_to_ansi256(0, 255, 0)
assert 34 <= color <= 46
def test_rgb_to_ansi256_blue(self, effect):
color = effect._rgb_to_ansi256(0, 0, 255)
assert 20 <= color <= 33
def test_configure_updates_params(self, effect):
config = EffectConfig(
enabled=True,
intensity=1.0,
params={"r": 100, "g": 150, "b": 200, "a": 0.8},
)
effect.configure(config)
assert effect.config.params["r"] == 100
assert effect.config.params["g"] == 150
assert effect.config.params["b"] == 200
assert effect.config.params["a"] == 0.8
def test_clamp_rgb_values(self, effect_with_params, mock_context):
effect_with_params.config.params["r"] = 300
effect_with_params.config.params["g"] = -10
effect_with_params.config.params["b"] = 1.5
buf = ["test"]
result = effect_with_params.process(buf, mock_context)
assert "\033[" in result[0]
def test_clamp_alpha_above_one(self, effect_with_params, mock_context):
effect_with_params.config.params["a"] = 1.5
buf = ["test"]
result = effect_with_params.process(buf, mock_context)
assert "\033[" in result[0]
def test_preserves_empty_lines(self, effect_with_params, mock_context):
buf = ["hello", "", "world"]
result = effect_with_params.process(buf, mock_context)
assert result[1] == ""
def test_inlet_types_includes_text_buffer(self, effect):
from engine.pipeline.core import DataType
assert DataType.TEXT_BUFFER in effect.inlet_types
def test_outlet_types_includes_text_buffer(self, effect):
from engine.pipeline.core import DataType
assert DataType.TEXT_BUFFER in effect.outlet_types