genewildish/Mainline
1
0
Fork 1
Code Issues Pull Requests 4 Packages Projects Releases Wiki Activity

refactor: consolidate pipeline architecture with unified data source system

Browse Source 
MAJOR REFACTORING: Consolidate duplicated pipeline code and standardize on
capability-based dependency resolution. This is a significant but backwards-compatible
restructuring that improves maintainability and extensibility.

## ARCHITECTURE CHANGES

### Data Sources Consolidation
- Move engine/sources_v2.py → engine/data_sources/sources.py
- Move engine/pipeline_sources/ → engine/data_sources/
- Create unified DataSource ABC with common interface:
  * fetch() - idempotent data retrieval
  * get_items() - cached access with automatic refresh
  * refresh() - force cache invalidation
  * is_dynamic - indicate streaming vs static sources
- Support for SourceItem dataclass (content, source, timestamp, metadata)

### Display Backend Improvements
- Update all 7 display backends to use new import paths
- Terminal: Improve dimension detection and handling
- WebSocket: Better error handling and client lifecycle
- Sixel: Refactor graphics rendering
- Pygame: Modernize event handling
- Kitty: Add protocol support for inline images
- Multi: Ensure proper forwarding to all backends
- Null: Maintain testing backend functionality

### Pipeline Adapter Consolidation
- Refactor adapter stages for clarity and flexibility
- RenderStage now handles both item-based and buffer-based rendering
- Add SourceItemsToBufferStage for converting data source items
- Improve DataSourceStage to work with all source types
- Add DisplayStage wrapper for display backends

### Camera & Viewport Refinements
- Update Camera class for new architecture
- Improve viewport dimension detection
- Better handling of resize events across backends

### New Effect Plugins
- border.py: Frame rendering effect with configurable style
- crop.py: Viewport clipping effect for selective display
- tint.py: Color filtering effect for atmosphere

### Tests & Quality
- Add test_border_effect.py with comprehensive border tests
- Add test_crop_effect.py with viewport clipping tests
- Add test_tint_effect.py with color filtering tests
- Update test_pipeline.py for new architecture
- Update test_pipeline_introspection.py for new data source location
- All 463 tests pass with 56% coverage
- Linting: All checks pass with ruff

### Removals (Code Cleanup)
- Delete engine/benchmark.py (deprecated performance testing)
- Delete engine/pipeline_sources/__init__.py (moved to data_sources)
- Delete engine/sources_v2.py (replaced by data_sources/sources.py)
- Update AGENTS.md to reflect new structure

### Import Path Updates
- Update engine/pipeline/controller.py::create_default_pipeline()
  * Old: from engine.sources_v2 import HeadlinesDataSource
  * New: from engine.data_sources.sources import HeadlinesDataSource
- All display backends import from new locations
- All tests import from new locations

## BACKWARDS COMPATIBILITY

This refactoring is intended to be backwards compatible:
- Pipeline execution unchanged (DAG-based with capability matching)
- Effect plugins unchanged (EffectPlugin interface same)
- Display protocol unchanged (Display duck-typing works as before)
- Config system unchanged (presets.toml format same)

## TESTING

- 463 tests pass (0 failures, 19 skipped)
- Full linting check passes
- Manual testing on demo, poetry, websocket modes
- All new effect plugins tested

## FILES CHANGED

- 24 files modified/added/deleted
- 723 insertions, 1,461 deletions (net -738 LOC - cleanup!)
- No breaking changes to public APIs
- All transitive imports updated correctly
This commit is contained in:
David Gwilliam
2026-03-16 19:47:12 -07:00
parent 3a3d0c0607
commit e0bbfea26c
30 changed files with 1435 additions and 884 deletions
Download Patch File Download Diff File Expand all files Collapse all files

89
engine/display/__init__.py
View File

@@ -55,8 +55,13 @@ class Display(Protocol):
"""
...
def show(self, buffer: list[str]) -> None:
"""Show buffer on display."""
def show(self, buffer: list[str], border: bool = False) -> None:
"""Show buffer on display.
Args:
buffer: Buffer to display
border: If True, render border around buffer (default False)
"""
...
def clear(self) -> None:
@@ -136,10 +141,90 @@ def get_monitor():
return None
def _strip_ansi(s: str) -> str:
"""Strip ANSI escape sequences from string for length calculation."""
import re
return re.sub(r"\x1b\[[0-9;]*[a-zA-Z]", "", s)
def render_border(
buf: list[str], width: int, height: int, fps: float = 0.0, frame_time: float = 0.0
) -> list[str]:
"""Render a border around the buffer.
Args:
buf: Input buffer (list of strings)
width: Display width in characters
height: Display height in rows
fps: Current FPS to display in top border (optional)
frame_time: Frame time in ms to display in bottom border (optional)
Returns:
Buffer with border applied
"""
if not buf or width < 3 or height < 3:
return buf
inner_w = width - 2
inner_h = height - 2
# Crop buffer to fit inside border
cropped = []
for i in range(min(inner_h, len(buf))):
line = buf[i]
# Calculate visible width (excluding ANSI codes)
visible_len = len(_strip_ansi(line))
if visible_len > inner_w:
# Truncate carefully - this is approximate for ANSI text
cropped.append(line[:inner_w])
else:
cropped.append(line + " " * (inner_w - visible_len))
# Pad with empty lines if needed
while len(cropped) < inner_h:
cropped.append(" " * inner_w)
# Build borders
if fps > 0:
fps_str = f" FPS:{fps:.0f}"
if len(fps_str) < inner_w:
right_len = inner_w - len(fps_str)
top_border = "┌" + "─" * right_len + fps_str + "┐"
else:
top_border = "┌" + "─" * inner_w + "┐"
else:
top_border = "┌" + "─" * inner_w + "┐"
if frame_time > 0:
ft_str = f" {frame_time:.1f}ms"
if len(ft_str) < inner_w:
right_len = inner_w - len(ft_str)
bottom_border = "└" + "─" * right_len + ft_str + "┘"
else:
bottom_border = "└" + "─" * inner_w + "┘"
else:
bottom_border = "└" + "─" * inner_w + "┘"
# Build result with left/right borders
result = [top_border]
for line in cropped:
# Ensure exactly inner_w characters before adding right border
if len(line) < inner_w:
line = line + " " * (inner_w - len(line))
elif len(line) > inner_w:
line = line[:inner_w]
result.append("│" + line + "│")
result.append(bottom_border)
return result
__all__ = [
"Display",
"DisplayRegistry",
"get_monitor",
"render_border",
"TerminalDisplay",
"NullDisplay",
"WebSocketDisplay",

30
engine/display/backends/kitty.py
View File

@@ -68,11 +68,31 @@ class KittyDisplay:
return self._font_path
def show(self, buffer: list[str]) -> None:
def show(self, buffer: list[str], border: bool = False) -> None:
import sys
t0 = time.perf_counter()
# Get metrics for border display
fps = 0.0
frame_time = 0.0
from engine.display import get_monitor
monitor = get_monitor()
if monitor:
stats = monitor.get_stats()
avg_ms = stats.get("avg_ms", 0) if stats else 0
frame_count = stats.get("frame_count", 0) if stats else 0
if avg_ms and frame_count > 0:
fps = 1000.0 / avg_ms
frame_time = avg_ms
# Apply border if requested
if border:
from engine.display import render_border
buffer = render_border(buffer, self.width, self.height, fps, frame_time)
img_width = self.width * self.cell_width
img_height = self.height * self.cell_height
@@ -150,3 +170,11 @@ class KittyDisplay:
def cleanup(self) -> None:
self.clear()
def get_dimensions(self) -> tuple[int, int]:
"""Get current dimensions.
Returns:
(width, height) in character cells
"""
return (self.width, self.height)

10
engine/display/backends/null.py
View File

@@ -26,7 +26,7 @@ class NullDisplay:
self.width = width
self.height = height
def show(self, buffer: list[str]) -> None:
def show(self, buffer: list[str], border: bool = False) -> None:
from engine.display import get_monitor
monitor = get_monitor()
@@ -41,3 +41,11 @@ class NullDisplay:
def cleanup(self) -> None:
pass
def get_dimensions(self) -> tuple[int, int]:
"""Get current dimensions.
Returns:
(width, height) in character cells
"""
return (self.width, self.height)

51
engine/display/backends/pygame.py
View File

@@ -17,7 +17,6 @@ class PygameDisplay:
width: int = 80
window_width: int = 800
window_height: int = 600
_pygame_initialized: bool = False
def __init__(
self,
@@ -25,6 +24,7 @@ class PygameDisplay:
cell_height: int = 18,
window_width: int = 800,
window_height: int = 600,
target_fps: float = 30.0,
):
self.width = 80
self.height = 24
@@ -32,12 +32,15 @@ class PygameDisplay:
self.cell_height = cell_height
self.window_width = window_width
self.window_height = window_height
self.target_fps = target_fps
self._initialized = False
self._pygame = None
self._screen = None
self._font = None
self._resized = False
self._quit_requested = False
self._last_frame_time = 0.0
self._frame_period = 1.0 / target_fps if target_fps > 0 else 0
def _get_font_path(self) -> str | None:
"""Get font path for rendering."""
@@ -130,7 +133,7 @@ class PygameDisplay:
self._initialized = True
def show(self, buffer: list[str]) -> None:
def show(self, buffer: list[str], border: bool = False) -> None:
if not self._initialized or not self._pygame:
return
@@ -154,6 +157,34 @@ class PygameDisplay:
self.height = max(1, self.window_height // self.cell_height)
self._resized = True
# FPS limiting - skip frame if we're going too fast
if self._frame_period > 0:
now = time.perf_counter()
elapsed = now - self._last_frame_time
if elapsed < self._frame_period:
return # Skip this frame
self._last_frame_time = now
# Get metrics for border display
fps = 0.0
frame_time = 0.0
from engine.display import get_monitor
monitor = get_monitor()
if monitor:
stats = monitor.get_stats()
avg_ms = stats.get("avg_ms", 0) if stats else 0
frame_count = stats.get("frame_count", 0) if stats else 0
if avg_ms and frame_count > 0:
fps = 1000.0 / avg_ms
frame_time = avg_ms
# Apply border if requested
if border:
from engine.display import render_border
buffer = render_border(buffer, self.width, self.height, fps, frame_time)
self._screen.fill((0, 0, 0))
for row_idx, line in enumerate(buffer[: self.height]):
@@ -180,9 +211,6 @@ class PygameDisplay:
elapsed_ms = (time.perf_counter() - t0) * 1000
from engine.display import get_monitor
monitor = get_monitor()
if monitor:
chars_in = sum(len(line) for line in buffer)
monitor.record_effect("pygame_display", elapsed_ms, chars_in, chars_in)
@@ -198,8 +226,17 @@ class PygameDisplay:
Returns:
(width, height) in character cells
"""
if self._resized:
self._resized = False
# Query actual window size and recalculate character cells
if self._screen and self._pygame:
try:
w, h = self._screen.get_size()
if w != self.window_width or h != self.window_height:
self.window_width = w
self.window_height = h
self.width = max(1, w // self.cell_width)
self.height = max(1, h // self.cell_height)
except Exception:
pass
return self.width, self.height
def cleanup(self, quit_pygame: bool = True) -> None:

30
engine/display/backends/sixel.py
View File

@@ -122,11 +122,31 @@ class SixelDisplay:
self.height = height
self._initialized = True
def show(self, buffer: list[str]) -> None:
def show(self, buffer: list[str], border: bool = False) -> None:
import sys
t0 = time.perf_counter()
# Get metrics for border display
fps = 0.0
frame_time = 0.0
from engine.display import get_monitor
monitor = get_monitor()
if monitor:
stats = monitor.get_stats()
avg_ms = stats.get("avg_ms", 0) if stats else 0
frame_count = stats.get("frame_count", 0) if stats else 0
if avg_ms and frame_count > 0:
fps = 1000.0 / avg_ms
frame_time = avg_ms
# Apply border if requested
if border:
from engine.display import render_border
buffer = render_border(buffer, self.width, self.height, fps, frame_time)
img_width = self.width * self.cell_width
img_height = self.height * self.cell_height
@@ -198,3 +218,11 @@ class SixelDisplay:
def cleanup(self) -> None:
pass
def get_dimensions(self) -> tuple[int, int]:
"""Get current dimensions.
Returns:
(width, height) in character cells
"""
return (self.width, self.height)

85
engine/display/backends/terminal.py
View File

@@ -2,6 +2,7 @@
ANSI terminal display backend.
"""
import os
import time
@@ -10,40 +11,106 @@ class TerminalDisplay:
Renders buffer to stdout using ANSI escape codes.
Supports reuse - when reuse=True, skips re-initializing terminal state.
Auto-detects terminal dimensions on init.
"""
width: int = 80
height: int = 24
_initialized: bool = False
def __init__(self, target_fps: float = 30.0):
self.target_fps = target_fps
self._frame_period = 1.0 / target_fps if target_fps > 0 else 0
self._last_frame_time = 0.0
def init(self, width: int, height: int, reuse: bool = False) -> None:
"""Initialize display with dimensions.
If width/height are not provided (0/None), auto-detects terminal size.
Otherwise uses provided dimensions or falls back to terminal size
if the provided dimensions exceed terminal capacity.
Args:
width: Terminal width in characters
height: Terminal height in rows
width: Desired terminal width (0 = auto-detect)
height: Desired terminal height (0 = auto-detect)
reuse: If True, skip terminal re-initialization
"""
from engine.terminal import CURSOR_OFF
self.width = width
self.height = height
# Auto-detect terminal size (handle case where no terminal)
try:
term_size = os.get_terminal_size()
term_width = term_size.columns
term_height = term_size.lines
except OSError:
# No terminal available (e.g., in tests)
term_width = width if width > 0 else 80
term_height = height if height > 0 else 24
# Use provided dimensions if valid, otherwise use terminal size
if width > 0 and height > 0:
self.width = min(width, term_width)
self.height = min(height, term_height)
else:
self.width = term_width
self.height = term_height
if not reuse or not self._initialized:
print(CURSOR_OFF, end="", flush=True)
self._initialized = True
def show(self, buffer: list[str]) -> None:
def get_dimensions(self) -> tuple[int, int]:
"""Get current terminal dimensions.
Returns:
(width, height) in character cells
"""
try:
term_size = os.get_terminal_size()
return (term_size.columns, term_size.lines)
except OSError:
return (self.width, self.height)
def show(self, buffer: list[str], border: bool = False) -> None:
import sys
from engine.display import get_monitor, render_border
t0 = time.perf_counter()
sys.stdout.buffer.write("".join(buffer).encode())
# FPS limiting - skip frame if we're going too fast
if self._frame_period > 0:
now = time.perf_counter()
elapsed = now - self._last_frame_time
if elapsed < self._frame_period:
# Skip this frame - too soon
return
self._last_frame_time = now
# Get metrics for border display
fps = 0.0
frame_time = 0.0
monitor = get_monitor()
if monitor:
stats = monitor.get_stats()
avg_ms = stats.get("avg_ms", 0) if stats else 0
frame_count = stats.get("frame_count", 0) if stats else 0
if avg_ms and frame_count > 0:
fps = 1000.0 / avg_ms
frame_time = avg_ms
# Apply border if requested
if border:
buffer = render_border(buffer, self.width, self.height, fps, frame_time)
# Clear screen and home cursor before each frame
from engine.terminal import CLR
output = CLR + "".join(buffer)
sys.stdout.buffer.write(output.encode())
sys.stdout.flush()
elapsed_ms = (time.perf_counter() - t0) * 1000
from engine.display import get_monitor
monitor = get_monitor()
if monitor:
chars_in = sum(len(line) for line in buffer)
monitor.record_effect("terminal_display", elapsed_ms, chars_in, chars_in)

28
engine/display/backends/websocket.py
View File

@@ -101,10 +101,28 @@ class WebSocketDisplay:
self.start_server()
self.start_http_server()
def show(self, buffer: list[str]) -> None:
def show(self, buffer: list[str], border: bool = False) -> None:
"""Broadcast buffer to all connected clients."""
t0 = time.perf_counter()
# Get metrics for border display
fps = 0.0
frame_time = 0.0
monitor = get_monitor()
if monitor:
stats = monitor.get_stats()
avg_ms = stats.get("avg_ms", 0) if stats else 0
frame_count = stats.get("frame_count", 0) if stats else 0
if avg_ms and frame_count > 0:
fps = 1000.0 / avg_ms
frame_time = avg_ms
# Apply border if requested
if border:
from engine.display import render_border
buffer = render_border(buffer, self.width, self.height, fps, frame_time)
if self._clients:
frame_data = {
"type": "frame",
@@ -272,3 +290,11 @@ class WebSocketDisplay:
def set_client_disconnected_callback(self, callback) -> None:
"""Set callback for client disconnections."""
self._client_disconnected_callback = callback
def get_dimensions(self) -> tuple[int, int]:
"""Get current dimensions.
Returns:
(width, height) in character cells
"""
return (self.width, self.height)