The old engine/pipeline/core.py file was removed as part of the Sideline/Mainline split.
All imports that referenced engine.pipeline.core have been updated to use engine.pipeline
which re-exports from sideline.pipeline.core.
This ensures consistency and avoids duplicate DataType enum instances.
- Change line 477 in controller.py to use self.config.display or "terminal"
- Previously hardcoded "terminal" ignored config and CLI arguments
- Now auto-injection respects the validated display configuration
fix(app): Add warnings for auto-selected display
- Add warning in pipeline_runner.py when --display not specified
- Add warning in main.py when --pipeline-display not specified
- Both warnings suggest using null display for headless mode
feat(completion): Add bash/zsh/fish completion scripts
- completion/mainline-completion.bash - bash completion
- completion/mainline-completion.zsh - zsh completion
- completion/mainline-completion.fish - fish completion
- Provides completions for --display, --pipeline-source, --pipeline-effects,
--pipeline-camera, --preset, --theme, --viewport, and other flags
- Add can_hot_swap() function to Pipeline class
- Add cleanup_stage() method to Pipeline class
- Fix remove_stage() to rebuild execution order after removal
- Extend ui_panel.execute_command() with docstrings for mutation commands
- Update WebSocket handler to support pipeline mutation commands
- Add _handle_pipeline_mutation() function for command routing
- Add comprehensive integration tests in test_pipeline_mutation_commands.py
- Update AGENTS.md with mutation API documentation
Issue: #35 (Pipeline Mutation API)
Acceptance criteria met:
- ✅ can_hot_swap() checker for stage compatibility
- ✅ cleanup_stage() cleans up specific stages
- ✅ remove_stage_safe() rebuilds execution order (via remove_stage)
- ✅ Unit tests for all operations
- ✅ Integration with WebSocket commands
- ✅ Documentation in AGENTS.md
- Implements pipeline hot-rebuild with state preservation (issue #43)
- Adds auto-injection of MVP stages for missing capabilities
- Adds radial camera mode for polar coordinate scanning
- Adds afterimage and motionblur effects using framebuffer history
- Adds comprehensive acceptance tests for camera modes and pipeline rebuild
- Updates presets.toml with new effect configurations
Related to: #35 (Pipeline Mutation API epic)
Closes: #43, #44, #45
- Add save_state/restore_state methods to CameraStage
- Add save_state/restore_state methods to DisplayStage
- Extend Pipeline._copy_stage_state() to preserve camera/display state
- Add save_state/restore_state methods to UIPanel for UI state preservation
- Update pipeline_runner to preserve UI state across preset changes
Camera state preserved:
- Position (x, y)
- Mode (feed, scroll, horizontal, etc.)
- Speed, zoom, canvas dimensions
- Internal timing state
Display state preserved:
- Initialization status
- Dimensions
- Reuse flag for display reinitialization
UI Panel state preserved:
- Stage enabled/disabled status
- Parameter values
- Selected stage and focused parameter
- Scroll position
This enables manual/event-driven rebuilds when inlet-outlet connections change,
while preserving all relevant state across pipeline mutations.
- Replace estimate_block_height (PIL-based) with estimate_simple_height (word wrap)
- Update viewport filter tests to match new height-based filtering (~4 items vs 24)
- Fix CI task duplication in mise.toml (remove redundant depends)
Closes#38Closes#36
## Summary
Fixed critical performance issue where demo/poetry presets would hang for 10+ seconds due to FontStage rendering all 1438+ headline items instead of just the visible ~5 items.
## Changes
### Core Fix: ViewportFilterStage
- New pipeline stage that filters items to only those fitting in the viewport
- Reduces 1438 items → ~5 items (288x reduction) before FontStage
- Prevents expensive PIL font rendering operations on items that won't be displayed
- Located: engine/pipeline/adapters.py:348-403
### Pipeline Integration
- Updated app.py to add ViewportFilterStage before FontStage for headlines/poetry sources
- Ensures correct data flow: source → viewport_filter → font → camera → effects → display
- ViewportFilterStage depends on 'source' capability, providing pass-through filtering
### Display Protocol Enhancement
- Added is_quit_requested() and clear_quit_request() method signatures to Display protocol
- Documented as optional methods for backends supporting keyboard input
- Already implemented by pygame backend, now formally part of protocol
### Debug Infrastructure
- Added MAINLINE_DEBUG_DATAFLOW environment variable logging throughout pipeline
- Logs stage input/output types and data sizes to stderr (when flag enabled)
- Verified working: 1438 → 5 item reduction shown in debug output
### Performance Testing
- Added pytest-benchmark (v5.2.3) as dev dependency for statistical benchmarking
- Created comprehensive performance regression tests (tests/test_performance_regression.py)
- Tests verify:
- ViewportFilterStage filters 2000 items efficiently (<1ms)
- FontStage processes filtered items quickly (<50ms)
- 288x performance improvement ratio maintained
- Pipeline doesn't hang with large datasets
- All 523 tests passing, including 7 new performance tests
## Performance Impact
**Before:** FontStage renders all 1438 items per frame → 10+ second hang
**After:** FontStage renders ~5 items per frame → sub-second execution
Real-world impact: Demo preset now responsive and usable with news sources.
## Testing
- Unit tests: 523 passed, 16 skipped
- Regression tests: Catch performance degradation with large datasets
- E2E verification: Debug logging confirms correct pipeline flow
- Benchmark suite: Statistical performance tracking enabled
Critical fix for display rendering:
1. DisplayStage Missing Dependency
- DisplayStage had empty dependencies, causing it to execute before render
- No data was reaching the display output
- Fix: Add 'render.output' dependency so display comes after render stages
- Now proper execution order: source → render → display
2. create_default_pipeline Missing Render Stage
- Default pipeline only had source and display, no render stage between
- Would fail validation with 'Missing render.output' capability
- Fix: Add SourceItemsToBufferStage to convert items to text buffer
- Now complete data flow: source → render → display
3. Updated Test Expectations
- test_display_stage_dependencies now expects 'render.output' dependency
Result: Display backends (pygame, terminal, websocket) now receive proper
rendered text buffers and can display output correctly.
Tests: All 502 tests passing
MAJOR REFACTORING: Consolidate duplicated pipeline code and standardize on
capability-based dependency resolution. This is a significant but backwards-compatible
restructuring that improves maintainability and extensibility.
## ARCHITECTURE CHANGES
### Data Sources Consolidation
- Move engine/sources_v2.py → engine/data_sources/sources.py
- Move engine/pipeline_sources/ → engine/data_sources/
- Create unified DataSource ABC with common interface:
* fetch() - idempotent data retrieval
* get_items() - cached access with automatic refresh
* refresh() - force cache invalidation
* is_dynamic - indicate streaming vs static sources
- Support for SourceItem dataclass (content, source, timestamp, metadata)
### Display Backend Improvements
- Update all 7 display backends to use new import paths
- Terminal: Improve dimension detection and handling
- WebSocket: Better error handling and client lifecycle
- Sixel: Refactor graphics rendering
- Pygame: Modernize event handling
- Kitty: Add protocol support for inline images
- Multi: Ensure proper forwarding to all backends
- Null: Maintain testing backend functionality
### Pipeline Adapter Consolidation
- Refactor adapter stages for clarity and flexibility
- RenderStage now handles both item-based and buffer-based rendering
- Add SourceItemsToBufferStage for converting data source items
- Improve DataSourceStage to work with all source types
- Add DisplayStage wrapper for display backends
### Camera & Viewport Refinements
- Update Camera class for new architecture
- Improve viewport dimension detection
- Better handling of resize events across backends
### New Effect Plugins
- border.py: Frame rendering effect with configurable style
- crop.py: Viewport clipping effect for selective display
- tint.py: Color filtering effect for atmosphere
### Tests & Quality
- Add test_border_effect.py with comprehensive border tests
- Add test_crop_effect.py with viewport clipping tests
- Add test_tint_effect.py with color filtering tests
- Update test_pipeline.py for new architecture
- Update test_pipeline_introspection.py for new data source location
- All 463 tests pass with 56% coverage
- Linting: All checks pass with ruff
### Removals (Code Cleanup)
- Delete engine/benchmark.py (deprecated performance testing)
- Delete engine/pipeline_sources/__init__.py (moved to data_sources)
- Delete engine/sources_v2.py (replaced by data_sources/sources.py)
- Update AGENTS.md to reflect new structure
### Import Path Updates
- Update engine/pipeline/controller.py::create_default_pipeline()
* Old: from engine.sources_v2 import HeadlinesDataSource
* New: from engine.data_sources.sources import HeadlinesDataSource
- All display backends import from new locations
- All tests import from new locations
## BACKWARDS COMPATIBILITY
This refactoring is intended to be backwards compatible:
- Pipeline execution unchanged (DAG-based with capability matching)
- Effect plugins unchanged (EffectPlugin interface same)
- Display protocol unchanged (Display duck-typing works as before)
- Config system unchanged (presets.toml format same)
## TESTING
- 463 tests pass (0 failures, 19 skipped)
- Full linting check passes
- Manual testing on demo, poetry, websocket modes
- All new effect plugins tested
## FILES CHANGED
- 24 files modified/added/deleted
- 723 insertions, 1,461 deletions (net -738 LOC - cleanup!)
- No breaking changes to public APIs
- All transitive imports updated correctly
- Add PipelineIntrospectionSource that renders live ASCII DAG with metrics
- Add PipelineMetricsSensor exposing pipeline performance as sensor values
- Add PipelineIntrospectionDemo controller with 3-phase animation:
- Phase 1: Toggle effects one at a time (3s each)
- Phase 2: LFO drives intensity default→max→min→default
- Phase 3: All effects with shared LFO (infinite loop)
- Add pipeline-inspect preset
- Add get_frame_times() to Pipeline for sparkline data
- Add tests for new components
- Update mise.toml with pipeline-inspect preset task
- Add DataType enum (SOURCE_ITEMS, TEXT_BUFFER, etc.)
- Add inlet_types and outlet_types to Stage
- Add _validate_types() for type checking at build time
- Update tests with proper type annotations
- Add TransformDataSource for filtering/mapping source items
- Add MetricsDataSource for rendering live pipeline metrics as ASCII art
- Fix display stage registration in StageRegistry
- Register sources with both class name and simple name aliases
- Fix DisplayStage.init() to pass reuse parameter
- Simplify create_default_pipeline to use DataSourceStage wrapper
- Set pygame as default display
- Remove old pipeline tasks from mise.toml
- Add tests for new pipeline architecture
