fix: Implement ViewportFilterStage to prevent FontStage performance regression with large datasets

## Summary

Fixed critical performance issue where demo/poetry presets would hang for 10+ seconds due to FontStage rendering all 1438+ headline items instead of just the visible ~5 items.

## Changes

### Core Fix: ViewportFilterStage
- New pipeline stage that filters items to only those fitting in the viewport
- Reduces 1438 items → ~5 items (288x reduction) before FontStage
- Prevents expensive PIL font rendering operations on items that won't be displayed
- Located: engine/pipeline/adapters.py:348-403

### Pipeline Integration
- Updated app.py to add ViewportFilterStage before FontStage for headlines/poetry sources
- Ensures correct data flow: source → viewport_filter → font → camera → effects → display
- ViewportFilterStage depends on 'source' capability, providing pass-through filtering

### Display Protocol Enhancement
- Added is_quit_requested() and clear_quit_request() method signatures to Display protocol
- Documented as optional methods for backends supporting keyboard input
- Already implemented by pygame backend, now formally part of protocol

### Debug Infrastructure
- Added MAINLINE_DEBUG_DATAFLOW environment variable logging throughout pipeline
- Logs stage input/output types and data sizes to stderr (when flag enabled)
- Verified working: 1438 → 5 item reduction shown in debug output

### Performance Testing
- Added pytest-benchmark (v5.2.3) as dev dependency for statistical benchmarking
- Created comprehensive performance regression tests (tests/test_performance_regression.py)
- Tests verify:
  - ViewportFilterStage filters 2000 items efficiently (<1ms)
  - FontStage processes filtered items quickly (<50ms)
  - 288x performance improvement ratio maintained
  - Pipeline doesn't hang with large datasets
- All 523 tests passing, including 7 new performance tests

## Performance Impact

**Before:** FontStage renders all 1438 items per frame → 10+ second hang
**After:** FontStage renders ~5 items per frame → sub-second execution

Real-world impact: Demo preset now responsive and usable with news sources.

## Testing

- Unit tests: 523 passed, 16 skipped
- Regression tests: Catch performance degradation with large datasets
- E2E verification: Debug logging confirms correct pipeline flow
- Benchmark suite: Statistical performance tracking enabled
This commit is contained in:
2026-03-16 22:43:53 -07:00
parent 10c1d057a9
commit 4c97cfe6aa
7 changed files with 468 additions and 1 deletions

View File

@@ -255,6 +255,18 @@ class Pipeline:
1. Execute all non-overlay stages in dependency order
2. Apply overlay stages on top (sorted by render_order)
"""
import os
import sys
debug = os.environ.get("MAINLINE_DEBUG_DATAFLOW") == "1"
if debug:
print(
f"[PIPELINE.execute] Starting with data type: {type(data).__name__ if data else 'None'}",
file=sys.stderr,
flush=True,
)
if not self._initialized:
self.build()
@@ -303,8 +315,30 @@ class Pipeline:
stage_start = time.perf_counter() if self._metrics_enabled else 0
try:
if debug:
data_info = type(current_data).__name__
if isinstance(current_data, list):
data_info += f"[{len(current_data)}]"
print(
f"[STAGE.{name}] Starting with: {data_info}",
file=sys.stderr,
flush=True,
)
current_data = stage.process(current_data, self.context)
if debug:
data_info = type(current_data).__name__
if isinstance(current_data, list):
data_info += f"[{len(current_data)}]"
print(
f"[STAGE.{name}] Completed, output: {data_info}",
file=sys.stderr,
flush=True,
)
except Exception as e:
if debug:
print(f"[STAGE.{name}] ERROR: {e}", file=sys.stderr, flush=True)
if not stage.optional:
return StageResult(
success=False,