forked from genewildish/Mainline
chore: remove deprecated docs and add skills library docs
- Delete LEGACY_CLEANUP_CHECKLIST.md, LEGACY_CODE_ANALYSIS.md, LEGACY_CODE_INDEX.md, SESSION_SUMMARY.md (superseded by wiki) - Add Skills Library section to AGENTS.md documenting MCP skills - Add uv to mise.toml tool versions
This commit is contained in:
192
AGENTS.md
192
AGENTS.md
@@ -219,3 +219,195 @@ Key files:
|
||||
- `engine/effects/types.py` - EffectPlugin ABC and dataclasses
|
||||
- `engine/display/backends/` - Display backend implementations
|
||||
- `engine/eventbus.py` - Thread-safe event system
|
||||
=======
|
||||
## Testing
|
||||
|
||||
Tests live in `tests/` and follow the pattern `test_*.py`.
|
||||
|
||||
Run all tests:
|
||||
```bash
|
||||
mise run test
|
||||
```
|
||||
|
||||
Run with coverage:
|
||||
```bash
|
||||
mise run test-cov
|
||||
```
|
||||
|
||||
The project uses pytest with strict marker enforcement. Test configuration is in `pyproject.toml` under `[tool.pytest.ini_options]`.
|
||||
|
||||
### Test Coverage Strategy
|
||||
|
||||
Current coverage: 56% (463 tests)
|
||||
|
||||
Key areas with lower coverage (acceptable for now):
|
||||
- **app.py** (8%): Main entry point - integration heavy, requires terminal
|
||||
- **scroll.py** (10%): Terminal-dependent rendering logic (unused)
|
||||
|
||||
Key areas with good coverage:
|
||||
- **display/backends/null.py** (95%): Easy to test headlessly
|
||||
- **display/backends/terminal.py** (96%): Uses mocking
|
||||
- **display/backends/multi.py** (100%): Simple forwarding logic
|
||||
- **effects/performance.py** (99%): Pure Python logic
|
||||
- **eventbus.py** (96%): Simple event system
|
||||
- **effects/controller.py** (95%): Effects command handling
|
||||
|
||||
Areas needing more tests:
|
||||
- **websocket.py** (48%): Network I/O, hard to test in CI
|
||||
- **ntfy.py** (50%): Network I/O, hard to test in CI
|
||||
- **mic.py** (61%): Audio I/O, hard to test in CI
|
||||
|
||||
Note: Terminal-dependent modules (scroll, layers render) are harder to test in CI.
|
||||
Performance regression tests are in `tests/test_benchmark.py` with `@pytest.mark.benchmark`.
|
||||
|
||||
## Architecture Notes
|
||||
|
||||
- **ntfy.py** - standalone notification poller with zero internal dependencies
|
||||
- **sensors/** - Sensor framework (MicSensor, OscillatorSensor) for real-time input
|
||||
- **eventbus.py** provides thread-safe event publishing for decoupled communication
|
||||
- **effects/** - plugin architecture with performance monitoring
|
||||
- The new pipeline architecture: source → render → effects → display
|
||||
|
||||
#### Canvas & Camera
|
||||
|
||||
- **Canvas** (`engine/canvas.py`): 2D rendering surface with dirty region tracking
|
||||
- **Camera** (`engine/camera.py`): Viewport controller for scrolling content
|
||||
|
||||
The Canvas tracks dirty regions automatically when content is written (via `put_region`, `put_text`, `fill`), enabling partial buffer updates for optimized effect processing.
|
||||
|
||||
### Pipeline Architecture
|
||||
|
||||
The new Stage-based pipeline architecture provides capability-based dependency resolution:
|
||||
|
||||
- **Stage** (`engine/pipeline/core.py`): Base class for pipeline stages
|
||||
- **Pipeline** (`engine/pipeline/controller.py`): Executes stages with capability-based dependency resolution
|
||||
- **StageRegistry** (`engine/pipeline/registry.py`): Discovers and registers stages
|
||||
- **Stage Adapters** (`engine/pipeline/adapters.py`): Wraps existing components as stages
|
||||
|
||||
#### Capability-Based Dependencies
|
||||
|
||||
Stages declare capabilities (what they provide) and dependencies (what they need). The Pipeline resolves dependencies using prefix matching:
|
||||
- `"source"` matches `"source.headlines"`, `"source.poetry"`, etc.
|
||||
- This allows flexible composition without hardcoding specific stage names
|
||||
|
||||
#### Sensor Framework
|
||||
|
||||
- **Sensor** (`engine/sensors/__init__.py`): Base class for real-time input sensors
|
||||
- **SensorRegistry**: Discovers available sensors
|
||||
- **SensorStage**: Pipeline adapter that provides sensor values to effects
|
||||
- **MicSensor** (`engine/sensors/mic.py`): Self-contained microphone input
|
||||
- **OscillatorSensor** (`engine/sensors/oscillator.py`): Test sensor for development
|
||||
- **PipelineMetricsSensor** (`engine/sensors/pipeline_metrics.py`): Exposes pipeline metrics as sensor values
|
||||
|
||||
Sensors support param bindings to drive effect parameters in real-time.
|
||||
|
||||
#### Pipeline Introspection
|
||||
|
||||
- **PipelineIntrospectionSource** (`engine/data_sources/pipeline_introspection.py`): Renders live ASCII visualization of pipeline DAG with metrics
|
||||
- **PipelineIntrospectionDemo** (`engine/pipeline/pipeline_introspection_demo.py`): 3-phase demo controller for effect animation
|
||||
|
||||
Preset: `pipeline-inspect` - Live pipeline introspection with DAG and performance metrics
|
||||
|
||||
#### Partial Update Support
|
||||
|
||||
Effect plugins can opt-in to partial buffer updates for performance optimization:
|
||||
- Set `supports_partial_updates = True` on the effect class
|
||||
- Implement `process_partial(buf, ctx, partial)` method
|
||||
- The `PartialUpdate` dataclass indicates which regions changed
|
||||
|
||||
### Preset System
|
||||
|
||||
Presets use TOML format (no external dependencies):
|
||||
|
||||
- Built-in: `engine/presets.toml`
|
||||
- User config: `~/.config/mainline/presets.toml`
|
||||
- Local override: `./presets.toml`
|
||||
|
||||
- **Preset loader** (`engine/pipeline/preset_loader.py`): Loads and validates presets
|
||||
- **PipelinePreset** (`engine/pipeline/presets.py`): Dataclass for preset configuration
|
||||
|
||||
Functions:
|
||||
- `validate_preset()` - Validate preset structure
|
||||
- `validate_signal_path()` - Detect circular dependencies
|
||||
- `generate_preset_toml()` - Generate skeleton preset
|
||||
|
||||
### Display System
|
||||
|
||||
- **Display abstraction** (`engine/display/`): swap display backends via the Display protocol
|
||||
- `display/backends/terminal.py` - ANSI terminal output
|
||||
- `display/backends/websocket.py` - broadcasts to web clients via WebSocket
|
||||
- `display/backends/sixel.py` - renders to Sixel graphics (pure Python, no C dependency)
|
||||
- `display/backends/null.py` - headless display for testing
|
||||
- `display/backends/multi.py` - forwards to multiple displays simultaneously
|
||||
- `display/__init__.py` - DisplayRegistry for backend discovery
|
||||
|
||||
- **WebSocket display** (`engine/display/backends/websocket.py`): real-time frame broadcasting to web browsers
|
||||
- WebSocket server on port 8765
|
||||
- HTTP server on port 8766 (serves HTML client)
|
||||
- Client at `client/index.html` with ANSI color parsing and fullscreen support
|
||||
|
||||
- **Display modes** (`--display` flag):
|
||||
- `terminal` - Default ANSI terminal output
|
||||
- `websocket` - Web browser display (requires websockets package)
|
||||
- `sixel` - Sixel graphics in supported terminals (iTerm2, mintty, etc.)
|
||||
- `both` - Terminal + WebSocket simultaneously
|
||||
|
||||
### Effect Plugin System
|
||||
|
||||
- **EffectPlugin ABC** (`engine/effects/types.py`): abstract base class for effects
|
||||
- All effects must inherit from EffectPlugin and implement `process()` and `configure()`
|
||||
- Runtime discovery via `effects_plugins/__init__.py` using `issubclass()` checks
|
||||
|
||||
- **EffectRegistry** (`engine/effects/registry.py`): manages registered effects
|
||||
- **EffectChain** (`engine/effects/chain.py`): chains effects in pipeline order
|
||||
|
||||
### Command & Control
|
||||
|
||||
- C&C uses separate ntfy topics for commands and responses
|
||||
- `NTFY_CC_CMD_TOPIC` - commands from cmdline.py
|
||||
- `NTFY_CC_RESP_TOPIC` - responses back to cmdline.py
|
||||
- Effects controller handles `/effects` commands (list, on/off, intensity, reorder, stats)
|
||||
|
||||
### Pipeline Documentation
|
||||
|
||||
The rendering pipeline is documented in `docs/PIPELINE.md` using Mermaid diagrams.
|
||||
|
||||
**IMPORTANT**: When making significant architectural changes to the rendering pipeline (new layers, effects, display backends), update `docs/PIPELINE.md` to reflect the changes:
|
||||
1. Edit `docs/PIPELINE.md` with the new architecture
|
||||
2. If adding new SVG diagrams, render them manually using an external tool (e.g., Mermaid Live Editor)
|
||||
3. Commit both the markdown and any new diagram files
|
||||
|
||||
## Skills Library
|
||||
|
||||
A skills library MCP server (`skills`) is available for capturing and tracking learned knowledge. Skills are stored in `~/.skills/`.
|
||||
|
||||
### Workflow
|
||||
|
||||
**Before starting work:**
|
||||
1. Run `skills_list_skills` to see available skills
|
||||
2. Use `skills_peek_skill({name: "skill-name"})` to preview relevant skills
|
||||
3. Use `skills_skill_slice({name: "skill-name", query: "your question"})` to get relevant sections
|
||||
|
||||
**While working:**
|
||||
- If a skill was wrong or incomplete: `skills_update_skill` → `skills_record_assessment` → `skills_report_outcome({quality: 1})`
|
||||
- If a skill worked correctly: `skills_report_outcome({quality: 4})` (normal) or `quality: 5` (perfect)
|
||||
|
||||
**End of session:**
|
||||
- Run `skills_reflect_on_session({context_summary: "what you did"})` to identify new skills to capture
|
||||
- Use `skills_create_skill` to add new skills
|
||||
- Use `skills_record_assessment` to score them
|
||||
|
||||
### Useful Tools
|
||||
- `skills_review_stale_skills()` - Skills due for review (negative days_until_due)
|
||||
- `skills_skills_report()` - Overview of entire collection
|
||||
- `skills_validate_skill({name: "skill-name"})` - Load skill for review with sources
|
||||
|
||||
### Agent Skills
|
||||
|
||||
This project also has Agent Skills (SKILL.md files) in `.opencode/skills/`. Use the `skill` tool to load them:
|
||||
- `skill({name: "mainline-architecture"})` - Pipeline stages, capability resolution
|
||||
- `skill({name: "mainline-effects"})` - How to add new effect plugins
|
||||
- `skill({name: "mainline-display"})` - Display backend implementation
|
||||
- `skill({name: "mainline-sources"})` - Adding new RSS feeds
|
||||
- `skill({name: "mainline-presets"})` - Creating pipeline presets
|
||||
- `skill({name: "mainline-sensors"})` - Sensor framework usage
|
||||
|
||||
@@ -1,239 +0,0 @@
|
||||
# Legacy Code Cleanup - Actionable Checklist
|
||||
|
||||
## Phase 1: Safe Removals (0 Risk, Run Immediately)
|
||||
|
||||
These modules have ZERO dependencies and can be removed without any testing:
|
||||
|
||||
### Files to Delete
|
||||
|
||||
```bash
|
||||
# Core modules (402 lines total)
|
||||
rm /home/dietpi/src/Mainline/engine/emitters.py (25 lines)
|
||||
rm /home/dietpi/src/Mainline/engine/beautiful_mermaid.py (4107 lines)
|
||||
rm /home/dietpi/src/Mainline/engine/pipeline_viz.py (364 lines)
|
||||
|
||||
# Test files (2145 bytes)
|
||||
rm /home/dietpi/src/Mainline/tests/test_emitters.py
|
||||
|
||||
# Configuration/cleanup
|
||||
# Remove from pipeline.py: introspect_pipeline_viz() method calls
|
||||
# Remove from pipeline.py: introspect_animation() references to pipeline_viz
|
||||
```
|
||||
|
||||
### Verification Commands
|
||||
|
||||
```bash
|
||||
# Verify emitters.py has zero references
|
||||
grep -r "from engine.emitters\|import.*emitters" /home/dietpi/src/Mainline --include="*.py" | grep -v "__pycache__" | grep -v ".venv"
|
||||
# Expected: NO RESULTS
|
||||
|
||||
# Verify beautiful_mermaid.py only used by pipeline_viz
|
||||
grep -r "beautiful_mermaid" /home/dietpi/src/Mainline --include="*.py" | grep -v "__pycache__" | grep -v ".venv"
|
||||
# Expected: Only one match in pipeline_viz.py
|
||||
|
||||
# Verify pipeline_viz.py has zero real usage
|
||||
grep -r "pipeline_viz\|CameraLarge\|PipelineIntrospection" /home/dietpi/src/Mainline --include="*.py" | grep -v "__pycache__" | grep -v ".venv" | grep -v "engine/pipeline_viz.py"
|
||||
# Expected: Only references in pipeline.py's introspection method
|
||||
```
|
||||
|
||||
### After Deletion - Cleanup Steps
|
||||
|
||||
1. Remove these lines from `engine/pipeline.py`:
|
||||
|
||||
```python
|
||||
# Remove method: introspect_pipeline_viz() (entire method)
|
||||
def introspect_pipeline_viz(self) -> None:
|
||||
# ... remove this entire method ...
|
||||
pass
|
||||
|
||||
# Remove method call from introspect():
|
||||
self.introspect_pipeline_viz()
|
||||
|
||||
# Remove import line:
|
||||
elif "pipeline_viz" in node.module or "CameraLarge" in node.name:
|
||||
```
|
||||
|
||||
2. Update imports in `engine/pipeline/__init__.py` if pipeline_viz is exported
|
||||
|
||||
3. Run test suite to verify:
|
||||
```bash
|
||||
mise run test
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Audit Required
|
||||
|
||||
### Action Items
|
||||
|
||||
#### 2.1 Pygame Backend Check
|
||||
|
||||
```bash
|
||||
# Find all preset definitions
|
||||
grep -r "display.*=.*['\"]pygame" /home/dietpi/src/Mainline --include="*.py" --include="*.toml"
|
||||
|
||||
# Search preset files
|
||||
grep -r "display.*pygame" /home/dietpi/src/Mainline/engine/presets.toml
|
||||
grep -r "pygame" /home/dietpi/src/Mainline/presets.toml
|
||||
|
||||
# If NO results: Safe to remove
|
||||
rm /home/dietpi/src/Mainline/engine/display/backends/pygame.py
|
||||
# And remove from DisplayRegistry.__init__: cls.register("pygame", PygameDisplay)
|
||||
# And remove import: from engine.display.backends.pygame import PygameDisplay
|
||||
|
||||
# If results exist: Keep the backend
|
||||
```
|
||||
|
||||
#### 2.2 Kitty Backend Check
|
||||
|
||||
```bash
|
||||
# Find all preset definitions
|
||||
grep -r "display.*=.*['\"]kitty" /home/dietpi/src/Mainline --include="*.py" --include="*.toml"
|
||||
|
||||
# Search preset files
|
||||
grep -r "display.*kitty" /home/dietpi/src/Mainline/engine/presets.toml
|
||||
grep -r "kitty" /home/dietpi/src/Mainline/presets.toml
|
||||
|
||||
# If NO results: Safe to remove
|
||||
rm /home/dietpi/src/Mainline/engine/display/backends/kitty.py
|
||||
# And remove from DisplayRegistry.__init__: cls.register("kitty", KittyDisplay)
|
||||
# And remove import: from engine.display.backends.kitty import KittyDisplay
|
||||
|
||||
# If results exist: Keep the backend
|
||||
```
|
||||
|
||||
#### 2.3 Animation Module Check
|
||||
|
||||
```bash
|
||||
# Search for actual usage of AnimationController, create_demo_preset, create_pipeline_preset
|
||||
grep -r "AnimationController\|create_demo_preset\|create_pipeline_preset" /home/dietpi/src/Mainline --include="*.py" | grep -v "animation.py" | grep -v "test_" | grep -v ".venv"
|
||||
|
||||
# If NO results: Safe to remove
|
||||
rm /home/dietpi/src/Mainline/engine/animation.py
|
||||
|
||||
# If results exist: Keep the module
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Known Future Removals (Don't Remove Yet)
|
||||
|
||||
These modules are marked deprecated and still in use. Plan to remove after their clients are migrated:
|
||||
|
||||
### Schedule for Removal
|
||||
|
||||
#### After scroll.py clients migrated:
|
||||
```bash
|
||||
rm /home/dietpi/src/Mainline/engine/scroll.py
|
||||
```
|
||||
|
||||
#### Consolidate legacy modules:
|
||||
```bash
|
||||
# After render.py functions are no longer called from adapters:
|
||||
# Move render.py to engine/legacy/render.py
|
||||
# Consolidate render.py with effects/legacy.py
|
||||
|
||||
# After layers.py functions are no longer called:
|
||||
# Move layers.py to engine/legacy/layers.py
|
||||
# Move effects/legacy.py functions alongside
|
||||
```
|
||||
|
||||
#### After legacy adapters are phased out:
|
||||
```bash
|
||||
rm /home/dietpi/src/Mainline/engine/pipeline/adapters.py (or move to legacy)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## How to Verify Changes
|
||||
|
||||
After making changes, run:
|
||||
|
||||
```bash
|
||||
# Run full test suite
|
||||
mise run test
|
||||
|
||||
# Run with coverage
|
||||
mise run test-cov
|
||||
|
||||
# Run linter
|
||||
mise run lint
|
||||
|
||||
# Check for import errors
|
||||
python3 -c "import engine.app; print('OK')"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary of File Changes
|
||||
|
||||
### Phase 1 Deletions (Safe)
|
||||
|
||||
| File | Lines | Purpose | Verify With |
|
||||
|------|-------|---------|------------|
|
||||
| engine/emitters.py | 25 | Unused protocols | `grep -r emitters` |
|
||||
| engine/beautiful_mermaid.py | 4107 | Unused diagram renderer | `grep -r beautiful_mermaid` |
|
||||
| engine/pipeline_viz.py | 364 | Unused visualization | `grep -r pipeline_viz` |
|
||||
| tests/test_emitters.py | 2145 bytes | Tests for emitters | Auto-removed with module |
|
||||
|
||||
### Phase 2 Conditional
|
||||
|
||||
| File | Size | Condition | Action |
|
||||
|------|------|-----------|--------|
|
||||
| engine/display/backends/pygame.py | 9185 | If not in presets | Delete or keep |
|
||||
| engine/display/backends/kitty.py | 5305 | If not in presets | Delete or keep |
|
||||
| engine/animation.py | 340 | If not used | Safe to delete |
|
||||
|
||||
### Phase 3 Future
|
||||
|
||||
| File | Lines | When | Action |
|
||||
|------|-------|------|--------|
|
||||
| engine/scroll.py | 156 | Deprecated | Plan removal |
|
||||
| engine/render.py | 274 | Still used | Consolidate later |
|
||||
| engine/layers.py | 272 | Still used | Consolidate later |
|
||||
|
||||
---
|
||||
|
||||
## Testing After Cleanup
|
||||
|
||||
1. **Unit Tests**: `mise run test`
|
||||
2. **Coverage Report**: `mise run test-cov`
|
||||
3. **Linting**: `mise run lint`
|
||||
4. **Manual Testing**: `mise run run` (run app in various presets)
|
||||
|
||||
### Expected Test Results After Phase 1
|
||||
|
||||
- No new test failures
|
||||
- test_emitters.py collection skipped (module removed)
|
||||
- All other tests pass
|
||||
- No import errors
|
||||
|
||||
---
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If issues arise after deletion:
|
||||
|
||||
```bash
|
||||
# Check git status
|
||||
git status
|
||||
|
||||
# Revert specific deletions
|
||||
git restore engine/emitters.py
|
||||
git restore engine/beautiful_mermaid.py
|
||||
# etc.
|
||||
|
||||
# Or full rollback
|
||||
git checkout HEAD -- engine/
|
||||
git checkout HEAD -- tests/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
- All Phase 1 deletions are verified to have ZERO usage
|
||||
- Phase 2 requires checking presets (can be done via grep)
|
||||
- Phase 3 items are actively used but marked for future removal
|
||||
- Keep test files synchronized with module deletions
|
||||
- Update AGENTS.md after Phase 1 completion
|
||||
@@ -1,286 +0,0 @@
|
||||
# Legacy & Dead Code Analysis - Mainline Codebase
|
||||
|
||||
## Executive Summary
|
||||
|
||||
The codebase contains **702 lines** of clearly marked legacy code spread across **4 main modules**, plus several candidate modules that may be unused. The legacy code primarily relates to the old rendering pipeline that has been superseded by the new Stage-based pipeline architecture.
|
||||
|
||||
---
|
||||
|
||||
## 1. MARKED DEPRECATED MODULES (Should Remove/Refactor)
|
||||
|
||||
### 1.1 `engine/scroll.py` (156 lines)
|
||||
- **Status**: DEPRECATED - Marked with deprecation notice
|
||||
- **Why**: Legacy rendering/orchestration code replaced by pipeline architecture
|
||||
- **Usage**: Used by legacy demo mode via scroll.stream()
|
||||
- **Dependencies**:
|
||||
- Imports: camera, display, layers, viewport, frame
|
||||
- Used by: scroll.py is only imported in tests and demo mode
|
||||
- **Risk**: LOW - Clean deprecation boundary
|
||||
- **Recommendation**: **SAFE TO REMOVE**
|
||||
- This is the main rendering loop orchestrator for the old system
|
||||
- All new code uses the Pipeline architecture
|
||||
- Demo mode is transitioning to pipeline presets
|
||||
- Consider keeping test_layers.py for testing layer functions
|
||||
|
||||
### 1.2 `engine/render.py` (274 lines)
|
||||
- **Status**: DEPRECATED - Marked with deprecation notice
|
||||
- **Why**: Legacy rendering code for font loading, text rasterization, gradient coloring
|
||||
- **Contains**:
|
||||
- `render_line()` - Renders text to terminal half-blocks using PIL
|
||||
- `big_wrap()` - Word-wrap text fitting
|
||||
- `lr_gradient()` - Left-to-right color gradients
|
||||
- `make_block()` - Assembles headline blocks
|
||||
- **Usage**:
|
||||
- layers.py imports: big_wrap, lr_gradient, lr_gradient_opposite
|
||||
- scroll.py conditionally imports make_block
|
||||
- adapters.py uses make_block
|
||||
- test_render.py tests these functions
|
||||
- **Risk**: MEDIUM - Used by legacy adapters and layers
|
||||
- **Recommendation**: **KEEP FOR NOW**
|
||||
- These functions are still used by adapters for legacy support
|
||||
- Could be moved to legacy submodule if cleanup needed
|
||||
- Consider marking functions individually as deprecated
|
||||
|
||||
### 1.3 `engine/layers.py` (272 lines)
|
||||
- **Status**: DEPRECATED - Marked with deprecation notice
|
||||
- **Why**: Legacy rendering layer logic for effects, overlays, firehose
|
||||
- **Contains**:
|
||||
- `render_ticker_zone()` - Renders ticker content
|
||||
- `render_firehose()` - Renders firehose effect
|
||||
- `render_message_overlay()` - Renders messages
|
||||
- `apply_glitch()` - Applies glitch effect
|
||||
- `process_effects()` - Legacy effect chain
|
||||
- `get_effect_chain()` - Access to legacy effect chain
|
||||
- **Usage**:
|
||||
- scroll.py imports multiple functions
|
||||
- effects/controller.py imports get_effect_chain as fallback
|
||||
- effects/__init__.py imports get_effect_chain as fallback
|
||||
- adapters.py imports render_firehose, render_ticker_zone
|
||||
- test_layers.py tests these functions
|
||||
- **Risk**: MEDIUM - Used as fallback in effects system
|
||||
- **Recommendation**: **KEEP FOR NOW**
|
||||
- Legacy effects system relies on this as fallback
|
||||
- Used by adapters for backwards compatibility
|
||||
- Mark individual functions as deprecated
|
||||
|
||||
### 1.4 `engine/animation.py` (340 lines)
|
||||
- **Status**: UNDEPRECATED but largely UNUSED
|
||||
- **Why**: Animation system with Clock, AnimationController, Preset classes
|
||||
- **Contains**:
|
||||
- Clock - High-resolution timer
|
||||
- AnimationController - Manages timed events and parameters
|
||||
- Preset - Bundles pipeline config + animation
|
||||
- Helper functions: create_demo_preset(), create_pipeline_preset()
|
||||
- Easing functions: linear_ease, ease_in_out, ease_out_bounce
|
||||
- **Usage**:
|
||||
- Documentation refers to it in pipeline.py docstrings
|
||||
- introspect_animation() method exists but generates no actual content
|
||||
- No actual imports of AnimationController found outside animation.py itself
|
||||
- Demo presets in animation.py are never called
|
||||
- PipelineParams dataclass is defined here but animation system never used
|
||||
- **Risk**: LOW - Isolated module with no real callers
|
||||
- **Recommendation**: **CONSIDER REMOVING**
|
||||
- This appears to be abandoned experimental code
|
||||
- The pipeline system doesn't actually use animation controllers
|
||||
- If animation is needed in future, should be redesigned
|
||||
- Safe to remove without affecting current functionality
|
||||
|
||||
---
|
||||
|
||||
## 2. COMPLETELY UNUSED MODULES (Safe to Remove)
|
||||
|
||||
### 2.1 `engine/emitters.py` (25 lines)
|
||||
- **Status**: UNUSED - Protocol definitions only
|
||||
- **Contains**: Three Protocol classes:
|
||||
- EventEmitter - Define subscribe/unsubscribe interface
|
||||
- Startable - Define start() interface
|
||||
- Stoppable - Define stop() interface
|
||||
- **Usage**: ZERO references found in codebase
|
||||
- **Risk**: NONE - Dead code
|
||||
- **Recommendation**: **SAFE TO REMOVE**
|
||||
- Protocol definitions are not used anywhere
|
||||
- EventBus uses its own implementation, doesn't inherit from these
|
||||
|
||||
### 2.2 `engine/beautiful_mermaid.py` (4107 lines!)
|
||||
- **Status**: UNUSED - Large ASCII renderer for Mermaid diagrams
|
||||
- **Why**: Pure Python Mermaid → ASCII renderer (ported from TypeScript)
|
||||
- **Usage**:
|
||||
- Only imported in pipeline_viz.py
|
||||
- pipeline_viz.py is not imported anywhere in codebase
|
||||
- Never called in production code
|
||||
- **Risk**: NONE - Dead code
|
||||
- **Recommendation**: **SAFE TO REMOVE**
|
||||
- Huge module (4000+ lines) with zero real usage
|
||||
- Only used by experimental pipeline_viz which itself is unused
|
||||
- Consider keeping as optional visualization tool if needed later
|
||||
|
||||
### 2.3 `engine/pipeline_viz.py` (364 lines)
|
||||
- **Status**: UNUSED - Pipeline visualization module
|
||||
- **Contains**: CameraLarge camera mode for pipeline visualization
|
||||
- **Usage**:
|
||||
- Only referenced in pipeline.py's introspect_pipeline_viz() method
|
||||
- This introspection method generates no actual output
|
||||
- Never instantiated or called in real code
|
||||
- **Risk**: NONE - Experimental dead code
|
||||
- **Recommendation**: **SAFE TO REMOVE**
|
||||
- Depends on beautiful_mermaid which is also unused
|
||||
- Remove together with beautiful_mermaid
|
||||
|
||||
---
|
||||
|
||||
## 3. UNUSED DISPLAY BACKENDS (Lower Priority)
|
||||
|
||||
These backends are registered in DisplayRegistry but may not be actively used:
|
||||
|
||||
### 3.1 `engine/display/backends/pygame.py` (9185 bytes)
|
||||
- **Status**: REGISTERED but potentially UNUSED
|
||||
- **Usage**: Registered in DisplayRegistry
|
||||
- **Last used in**: Demo mode (may have been replaced)
|
||||
- **Risk**: LOW - Backend system is pluggable
|
||||
- **Recommendation**: CHECK USAGE
|
||||
- Verify if any presets use "pygame" display
|
||||
- If not used, can remove
|
||||
- Otherwise keep as optional backend
|
||||
|
||||
### 3.2 `engine/display/backends/kitty.py` (5305 bytes)
|
||||
- **Status**: REGISTERED but potentially UNUSED
|
||||
- **Usage**: Registered in DisplayRegistry
|
||||
- **Last used in**: Kitty terminal graphics protocol
|
||||
- **Risk**: LOW - Backend system is pluggable
|
||||
- **Recommendation**: CHECK USAGE
|
||||
- Verify if any presets use "kitty" display
|
||||
- If not used, can remove
|
||||
- Otherwise keep as optional backend
|
||||
|
||||
### 3.3 `engine/display/backends/multi.py` (1137 bytes)
|
||||
- **Status**: REGISTERED and likely USED
|
||||
- **Usage**: MultiDisplay for simultaneous output
|
||||
- **Risk**: LOW - Simple wrapper
|
||||
- **Recommendation**: KEEP
|
||||
|
||||
---
|
||||
|
||||
## 4. TEST FILES THAT MAY BE OBSOLETE
|
||||
|
||||
### 4.1 `tests/test_emitters.py` (2145 bytes)
|
||||
- **Status**: ORPHANED
|
||||
- **Why**: Tests for unused emitters protocols
|
||||
- **Recommendation**: **SAFE TO REMOVE**
|
||||
- Remove with engine/emitters.py
|
||||
|
||||
### 4.2 `tests/test_render.py` (7628 bytes)
|
||||
- **Status**: POTENTIALLY USEFUL
|
||||
- **Why**: Tests for legacy render functions still used by adapters
|
||||
- **Recommendation**: **KEEP FOR NOW**
|
||||
- Keep while render.py functions are used
|
||||
|
||||
### 4.3 `tests/test_layers.py` (3717 bytes)
|
||||
- **Status**: POTENTIALLY USEFUL
|
||||
- **Why**: Tests for legacy layer functions
|
||||
- **Recommendation**: **KEEP FOR NOW**
|
||||
- Keep while layers.py functions are used
|
||||
|
||||
---
|
||||
|
||||
## 5. QUESTIONABLE PATTERNS & TECHNICAL DEBT
|
||||
|
||||
### 5.1 Legacy Effect Chain Fallback
|
||||
**Location**: `effects/controller.py`, `effects/__init__.py`
|
||||
|
||||
```python
|
||||
# Fallback to legacy effect chain if no new effects available
|
||||
try:
|
||||
from engine.layers import get_effect_chain as _chain
|
||||
except ImportError:
|
||||
_chain = None
|
||||
```
|
||||
|
||||
**Issue**: Dual effect system with implicit fallback
|
||||
**Recommendation**: Document or remove fallback path if not actually used
|
||||
|
||||
### 5.2 Deprecated ItemsStage Bootstrap
|
||||
**Location**: `pipeline/adapters.py` line 356-365
|
||||
|
||||
```python
|
||||
@deprecated("ItemsStage is deprecated. Use DataSourceStage with a DataSource instead.")
|
||||
class ItemsStage(Stage):
|
||||
"""Deprecated bootstrap mechanism."""
|
||||
```
|
||||
|
||||
**Issue**: Marked deprecated but still registered and potentially used
|
||||
**Recommendation**: Audit usage and remove if not needed
|
||||
|
||||
### 5.3 Legacy Tuple Conversion Methods
|
||||
**Location**: `engine/types.py`
|
||||
|
||||
```python
|
||||
def to_legacy_tuple(self) -> tuple[list[tuple], int, int]:
|
||||
"""Convert to legacy tuple format for backward compatibility."""
|
||||
```
|
||||
|
||||
**Issue**: Backward compatibility layer that may not be needed
|
||||
**Recommendation**: Check if actually used by legacy code
|
||||
|
||||
### 5.4 Frame Module (Minimal Usage)
|
||||
**Location**: `engine/frame.py`
|
||||
|
||||
**Status**: Appears minimal and possibly legacy
|
||||
**Recommendation**: Check what's actually using it
|
||||
|
||||
---
|
||||
|
||||
## SUMMARY TABLE
|
||||
|
||||
| Module | LOC | Status | Risk | Action |
|
||||
|--------|-----|--------|------|--------|
|
||||
| scroll.py | 156 | **REMOVE** | LOW | Delete - fully deprecated |
|
||||
| emitters.py | 25 | **REMOVE** | NONE | Delete - zero usage |
|
||||
| beautiful_mermaid.py | 4107 | **REMOVE** | NONE | Delete - zero usage |
|
||||
| pipeline_viz.py | 364 | **REMOVE** | NONE | Delete - zero usage |
|
||||
| animation.py | 340 | CONSIDER | LOW | Remove if not planned |
|
||||
| render.py | 274 | KEEP | MEDIUM | Still used by adapters |
|
||||
| layers.py | 272 | KEEP | MEDIUM | Still used by adapters |
|
||||
| pygame backend | 9185 | AUDIT | LOW | Check if used |
|
||||
| kitty backend | 5305 | AUDIT | LOW | Check if used |
|
||||
| test_emitters.py | 2145 | **REMOVE** | NONE | Delete with emitters.py |
|
||||
|
||||
---
|
||||
|
||||
## RECOMMENDED CLEANUP STRATEGY
|
||||
|
||||
### Phase 1: Safe Removals (No Dependencies)
|
||||
1. Delete `engine/emitters.py`
|
||||
2. Delete `tests/test_emitters.py`
|
||||
3. Delete `engine/beautiful_mermaid.py`
|
||||
4. Delete `engine/pipeline_viz.py`
|
||||
5. Clean up related deprecation code in `pipeline.py`
|
||||
|
||||
**Impact**: ~4500 lines of dead code removed
|
||||
**Risk**: NONE - verified zero usage
|
||||
|
||||
### Phase 2: Conditional Removals (Audit Required)
|
||||
1. Verify pygame and kitty backends are not used in any preset
|
||||
2. If unused, remove from DisplayRegistry and delete files
|
||||
3. Consider removing `engine/animation.py` if animation features not planned
|
||||
|
||||
### Phase 3: Legacy Module Migration (Future)
|
||||
1. Move render.py functions to legacy submodule if scroll.py is removed
|
||||
2. Consolidate layers.py with legacy effects
|
||||
3. Keep test files until legacy adapters are phased out
|
||||
4. Deprecate legacy adapters in favor of new pipeline stages
|
||||
|
||||
### Phase 4: Documentation
|
||||
1. Update AGENTS.md to document removal of legacy modules
|
||||
2. Document which adapters are for backwards compatibility
|
||||
3. Add migration guide for teams using old scroll API
|
||||
|
||||
---
|
||||
|
||||
## KEY METRICS
|
||||
|
||||
- **Total Dead Code Lines**: ~9000+ lines
|
||||
- **Safe to Remove Immediately**: ~4500 lines
|
||||
- **Conditional Removals**: ~10000+ lines (if backends/animation unused)
|
||||
- **Legacy But Needed**: ~700 lines (render.py + layers.py)
|
||||
- **Test Files for Dead Code**: ~2100 lines
|
||||
|
||||
@@ -1,153 +0,0 @@
|
||||
# Legacy Code Analysis - Document Index
|
||||
|
||||
This directory contains comprehensive analysis of legacy and dead code in the Mainline codebase.
|
||||
|
||||
## Quick Start
|
||||
|
||||
**Start here:** [LEGACY_CLEANUP_CHECKLIST.md](./LEGACY_CLEANUP_CHECKLIST.md)
|
||||
|
||||
This document provides step-by-step instructions for removing dead code in three phases:
|
||||
- **Phase 1**: Safe removals (~4,500 lines, zero risk)
|
||||
- **Phase 2**: Audit required (~14,000 lines)
|
||||
- **Phase 3**: Future migration plan
|
||||
|
||||
## Available Documents
|
||||
|
||||
### 1. LEGACY_CLEANUP_CHECKLIST.md (Action-Oriented)
|
||||
**Purpose**: Step-by-step cleanup procedures with verification commands
|
||||
|
||||
**Contains**:
|
||||
- Phase 1: Safe deletions with verification commands
|
||||
- Phase 2: Audit procedures for display backends
|
||||
- Phase 3: Future removal planning
|
||||
- Testing procedures after cleanup
|
||||
- Rollback procedures
|
||||
|
||||
**Start reading if you want to**: Execute cleanup immediately
|
||||
|
||||
### 2. LEGACY_CODE_ANALYSIS.md (Detailed Technical)
|
||||
**Purpose**: Comprehensive technical analysis with risk assessments
|
||||
|
||||
**Contains**:
|
||||
- Executive summary
|
||||
- Marked deprecated modules (scroll.py, render.py, layers.py)
|
||||
- Completely unused modules (emitters.py, beautiful_mermaid.py, pipeline_viz.py)
|
||||
- Unused display backends
|
||||
- Test file analysis
|
||||
- Technical debt patterns
|
||||
- Cleanup strategy across 4 phases
|
||||
- Key metrics and statistics
|
||||
|
||||
**Start reading if you want to**: Understand the technical details
|
||||
|
||||
## Key Findings Summary
|
||||
|
||||
### Dead Code Identified: ~9,000 lines
|
||||
|
||||
#### Category 1: UNUSED (Safe to delete immediately)
|
||||
- **engine/emitters.py** (25 lines) - Unused Protocol definitions
|
||||
- **engine/beautiful_mermaid.py** (4,107 lines) - Unused Mermaid ASCII renderer
|
||||
- **engine/pipeline_viz.py** (364 lines) - Unused visualization module
|
||||
- **tests/test_emitters.py** - Orphaned test file
|
||||
|
||||
**Total**: ~4,500 lines with ZERO risk
|
||||
|
||||
#### Category 2: DEPRECATED BUT ACTIVE (Keep for now)
|
||||
- **engine/scroll.py** (156 lines) - Legacy rendering orchestration
|
||||
- **engine/render.py** (274 lines) - Legacy font/gradient rendering
|
||||
- **engine/layers.py** (272 lines) - Legacy layer/effect rendering
|
||||
|
||||
**Total**: ~700 lines (still used for backwards compatibility)
|
||||
|
||||
#### Category 3: QUESTIONABLE (Consider removing)
|
||||
- **engine/animation.py** (340 lines) - Unused animation system
|
||||
|
||||
**Total**: ~340 lines (abandoned experimental code)
|
||||
|
||||
#### Category 4: POTENTIALLY UNUSED (Requires audit)
|
||||
- **engine/display/backends/pygame.py** (9,185 bytes)
|
||||
- **engine/display/backends/kitty.py** (5,305 bytes)
|
||||
|
||||
**Total**: ~14,000 bytes (check if presets use them)
|
||||
|
||||
## File Paths
|
||||
|
||||
### Recommended for Deletion (Phase 1)
|
||||
```
|
||||
/home/dietpi/src/Mainline/engine/emitters.py
|
||||
/home/dietpi/src/Mainline/engine/beautiful_mermaid.py
|
||||
/home/dietpi/src/Mainline/engine/pipeline_viz.py
|
||||
/home/dietpi/src/Mainline/tests/test_emitters.py
|
||||
```
|
||||
|
||||
### Keep for Now (Legacy Backwards Compatibility)
|
||||
```
|
||||
/home/dietpi/src/Mainline/engine/scroll.py
|
||||
/home/dietpi/src/Mainline/engine/render.py
|
||||
/home/dietpi/src/Mainline/engine/layers.py
|
||||
```
|
||||
|
||||
### Requires Audit (Phase 2)
|
||||
```
|
||||
/home/dietpi/src/Mainline/engine/display/backends/pygame.py
|
||||
/home/dietpi/src/Mainline/engine/display/backends/kitty.py
|
||||
```
|
||||
|
||||
## Recommended Reading Order
|
||||
|
||||
1. **First**: This file (overview)
|
||||
2. **Then**: LEGACY_CLEANUP_CHECKLIST.md (if you want to act immediately)
|
||||
3. **Or**: LEGACY_CODE_ANALYSIS.md (if you want to understand deeply)
|
||||
|
||||
## Key Statistics
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Total Dead Code | ~9,000 lines |
|
||||
| Safe to Remove (Phase 1) | ~4,500 lines |
|
||||
| Conditional Removals (Phase 2) | ~3,800 lines |
|
||||
| Legacy But Active (Phase 3) | ~700 lines |
|
||||
| Risk Level (Phase 1) | NONE |
|
||||
| Risk Level (Phase 2) | LOW |
|
||||
| Risk Level (Phase 3) | MEDIUM |
|
||||
|
||||
## Action Items
|
||||
|
||||
### Immediate (Phase 1 - 0 Risk)
|
||||
- [ ] Delete engine/emitters.py
|
||||
- [ ] Delete tests/test_emitters.py
|
||||
- [ ] Delete engine/beautiful_mermaid.py
|
||||
- [ ] Delete engine/pipeline_viz.py
|
||||
- [ ] Clean up pipeline.py introspection methods
|
||||
|
||||
### Short Term (Phase 2 - Low Risk)
|
||||
- [ ] Audit pygame backend usage
|
||||
- [ ] Audit kitty backend usage
|
||||
- [ ] Decide on animation.py
|
||||
|
||||
### Future (Phase 3 - Medium Risk)
|
||||
- [ ] Plan scroll.py migration
|
||||
- [ ] Consolidate render.py/layers.py
|
||||
- [ ] Deprecate legacy adapters
|
||||
|
||||
## How to Execute Cleanup
|
||||
|
||||
See [LEGACY_CLEANUP_CHECKLIST.md](./LEGACY_CLEANUP_CHECKLIST.md) for:
|
||||
- Exact deletion commands
|
||||
- Verification procedures
|
||||
- Testing procedures
|
||||
- Rollback procedures
|
||||
|
||||
## Questions?
|
||||
|
||||
Refer to the detailed analysis documents:
|
||||
- For specific module details: LEGACY_CODE_ANALYSIS.md
|
||||
- For how to delete: LEGACY_CLEANUP_CHECKLIST.md
|
||||
- For verification commands: LEGACY_CLEANUP_CHECKLIST.md (Phase 1 section)
|
||||
|
||||
---
|
||||
|
||||
**Analysis Date**: March 16, 2026
|
||||
**Codebase**: Mainline (Pipeline Architecture)
|
||||
**Legacy Code Found**: ~9,000 lines
|
||||
**Safe to Remove Now**: ~4,500 lines
|
||||
@@ -1,501 +0,0 @@
|
||||
# Session Summary: Phase 2, 3 & 4 Complete
|
||||
|
||||
**Date:** March 16, 2026
|
||||
**Duration:** Full session
|
||||
**Overall Achievement:** 126 new tests added, 5,296+ lines of legacy code cleaned up, RenderStage/ItemsStage removed, codebase modernized
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
This session accomplished four major phases of work:
|
||||
|
||||
1. **Phase 2: Test Coverage Improvements** - Added 67 comprehensive tests
|
||||
2. **Phase 3 (Early): Legacy Code Removal** - Removed 4,840 lines of dead code (Phases 1-2)
|
||||
3. **Phase 3 (Full): Legacy Module Migration** - Reorganized remaining legacy code into dedicated subsystem
|
||||
4. **Phase 4: Remove Deprecated Adapters** - Deleted RenderStage and ItemsStage, replaced with modern patterns
|
||||
|
||||
**Final Stats:**
|
||||
- Tests: 463 → 530 → 521 → 515 → 508 passing (508 core tests, 6 legacy failures pre-existing)
|
||||
- Core tests (non-legacy): 67 new tests added
|
||||
- Lines of code removed: 5,576 lines total (5,296 + 280 from Phase 4)
|
||||
- Legacy code properly organized in `engine/legacy/` and `tests/legacy/`
|
||||
- Deprecated adapters fully removed and replaced
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Remove Deprecated Adapters (Complete Refactor)
|
||||
|
||||
### Overview
|
||||
|
||||
Phase 4 completed the removal of two deprecated pipeline adapter classes:
|
||||
- **RenderStage** (124 lines) - Legacy rendering layer
|
||||
- **ItemsStage** (32 lines) - Bootstrap mechanism for pre-fetched items
|
||||
- **create_items_stage()** function (3 lines)
|
||||
|
||||
**Replacement Strategy:**
|
||||
- Created `ListDataSource` class (38 lines) to wrap arbitrary pre-fetched items
|
||||
- Updated app.py to use DataSourceStage + ListDataSource pattern
|
||||
- Removed 7 deprecated test methods
|
||||
|
||||
**Net Result:** 280 lines removed, 0 regressions, 508 core tests passing
|
||||
|
||||
### Phase 4.1: Add Deprecation Warnings (7c69086)
|
||||
**File:** `engine/pipeline/adapters.py`
|
||||
|
||||
Added DeprecationWarning to RenderStage.__init__():
|
||||
- Notifies developers that RenderStage uses legacy rendering code
|
||||
- Points to modern replacement (SourceItemsToBufferStage)
|
||||
- Prepares codebase for full removal
|
||||
|
||||
### Phase 4.2: Remove RenderStage Usage from app.py (3e73ea0)
|
||||
**File:** `engine/app.py`
|
||||
|
||||
Replaced RenderStage with SourceItemsToBufferStage:
|
||||
- Removed special-case logic for non-special sources
|
||||
- Simplified render pipeline - all sources use same modern path
|
||||
- All 11 app integration tests pass
|
||||
- No behavior change, only architecture improvement
|
||||
|
||||
### Phase 4.3: Delete Deprecated Classes (6fc3cbc)
|
||||
**Files:** `engine/pipeline/adapters.py`, `engine/data_sources/sources.py`, `tests/test_pipeline.py`
|
||||
|
||||
**Deletions:**
|
||||
1. **RenderStage class** (124 lines)
|
||||
- Used legacy engine.legacy.render and engine.legacy.layers
|
||||
- Replaced with SourceItemsToBufferStage + DataSourceStage pattern
|
||||
- Removed 4 test methods for RenderStage
|
||||
|
||||
2. **ItemsStage class** (32 lines)
|
||||
- Bootstrap mechanism for pre-fetched items
|
||||
- Removed 3 test methods for ItemsStage
|
||||
|
||||
3. **create_items_stage() function** (3 lines)
|
||||
- Helper to create ItemsStage instances
|
||||
- No longer needed
|
||||
|
||||
**Additions:**
|
||||
1. **ListDataSource class** (38 lines)
|
||||
- Wraps arbitrary pre-fetched items as DataSource
|
||||
- Allows items to be used with modern DataSourceStage
|
||||
- Simple implementation: stores items in constructor, returns in get_items()
|
||||
|
||||
**Test Removals:**
|
||||
- `test_render_stage_capabilities` - RenderStage-specific
|
||||
- `test_render_stage_dependencies` - RenderStage-specific
|
||||
- `test_render_stage_process` - RenderStage-specific
|
||||
- `test_datasource_stage_capabilities_match_render_deps` - RenderStage comparison
|
||||
- `test_items_stage` - ItemsStage-specific
|
||||
- `test_pipeline_with_items_and_effect` - ItemsStage usage
|
||||
- `test_pipeline_with_items_stage` - ItemsStage usage
|
||||
|
||||
**Impact:**
|
||||
- 159 lines deleted from adapters.py
|
||||
- 3 lines deleted from app.py
|
||||
- 38 lines added as ListDataSource
|
||||
- 7 test methods removed (expected deprecation)
|
||||
- **508 core tests pass** - no regressions
|
||||
|
||||
### Phase 4.4: Update Pipeline Introspection (d14f850)
|
||||
**File:** `engine/pipeline.py`
|
||||
|
||||
Removed documentation entries:
|
||||
- ItemsStage documentation removed
|
||||
- RenderStage documentation removed
|
||||
- Introspection DAG now only shows active stages
|
||||
|
||||
**Impact:**
|
||||
- Cleaner pipeline visualization
|
||||
- No confusion about deprecated adapters
|
||||
- 508 tests still passing
|
||||
|
||||
### Architecture Changes
|
||||
|
||||
**Before Phase 4:**
|
||||
```
|
||||
DataSourceStage
|
||||
↓
|
||||
(RenderStage - deprecated)
|
||||
↓
|
||||
SourceItemsToBufferStage
|
||||
↓
|
||||
DisplayStage
|
||||
|
||||
Bootstrap:
|
||||
(ItemsStage - deprecated, only for pre-fetched items)
|
||||
↓
|
||||
SourceItemsToBufferStage
|
||||
```
|
||||
|
||||
**After Phase 4:**
|
||||
```
|
||||
DataSourceStage (now wraps all sources, including ListDataSource)
|
||||
↓
|
||||
SourceItemsToBufferStage
|
||||
↓
|
||||
DisplayStage
|
||||
|
||||
Unified Pattern:
|
||||
ListDataSource wraps pre-fetched items
|
||||
↓
|
||||
DataSourceStage
|
||||
↓
|
||||
SourceItemsToBufferStage
|
||||
```
|
||||
|
||||
### Test Metrics
|
||||
|
||||
**Before Phase 4:**
|
||||
- 515 core tests passing
|
||||
- RenderStage had 4 dedicated tests
|
||||
- ItemsStage had 3 dedicated tests
|
||||
- create_items_stage() had related tests
|
||||
|
||||
**After Phase 4:**
|
||||
- 508 core tests passing
|
||||
- 7 deprecated tests removed (expected)
|
||||
- 19 tests skipped
|
||||
- 6 legacy tests failing (pre-existing, unrelated)
|
||||
- **Zero regressions** in modern code
|
||||
|
||||
### Code Quality
|
||||
|
||||
**Linting:** ✅ All checks pass
|
||||
- ruff format checks pass
|
||||
- ruff check passes
|
||||
- No style violations
|
||||
|
||||
**Testing:** ✅ Full suite passes
|
||||
```
|
||||
508 passed, 19 skipped, 6 failed (pre-existing legacy)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Test Coverage Improvements (67 new tests)
|
||||
|
||||
### Commit 1: Data Source Tests (d9c7138)
|
||||
**File:** `tests/test_data_sources.py` (220 lines, 19 tests)
|
||||
|
||||
Tests for:
|
||||
- `SourceItem` dataclass creation and metadata
|
||||
- `EmptyDataSource` - blank content generation
|
||||
- `HeadlinesDataSource` - RSS feed integration
|
||||
- `PoetryDataSource` - poetry source integration
|
||||
- `DataSource` base class interface
|
||||
|
||||
**Coverage Impact:**
|
||||
- `engine/data_sources/sources.py`: 34% → 39%
|
||||
|
||||
### Commit 2: Pipeline Adapter Tests (952b73c)
|
||||
**File:** `tests/test_adapters.py` (345 lines, 37 tests)
|
||||
|
||||
Tests for:
|
||||
- `DataSourceStage` - data source integration
|
||||
- `DisplayStage` - display backend integration
|
||||
- `PassthroughStage` - pass-through rendering
|
||||
- `SourceItemsToBufferStage` - content to buffer conversion
|
||||
- `EffectPluginStage` - effect application
|
||||
|
||||
**Coverage Impact:**
|
||||
- `engine/pipeline/adapters.py`: ~50% → 57%
|
||||
|
||||
### Commit 3: Fix App Integration Tests (28203ba)
|
||||
**File:** `tests/test_app.py` (fixed 7 tests)
|
||||
|
||||
Fixed issues:
|
||||
- Config mocking for PIPELINE_DIAGRAM flag
|
||||
- Proper display mock setup to prevent pygame window launch
|
||||
- Correct preset display backend expectations
|
||||
- All 11 app tests now passing
|
||||
|
||||
**Coverage Impact:**
|
||||
- `engine/app.py`: 0-8% → 67%
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Legacy Code Cleanup
|
||||
|
||||
### Phase 3.1: Dead Code Removal
|
||||
|
||||
**Commits:**
|
||||
- 5762d5e: Removed 4,500 lines of dead code
|
||||
- 0aa80f9: Removed 340 lines of unused animation.py
|
||||
|
||||
**Deleted:**
|
||||
- `engine/emitters.py` (25 lines) - unused Protocol definitions
|
||||
- `engine/beautiful_mermaid.py` (4,107 lines) - unused Mermaid ASCII renderer
|
||||
- `engine/pipeline_viz.py` (364 lines) - unused visualization module
|
||||
- `tests/test_emitters.py` (69 lines) - orphaned test file
|
||||
- `engine/animation.py` (340 lines) - abandoned experimental animation system
|
||||
- Cleanup of `engine/pipeline.py` introspection methods (25 lines)
|
||||
|
||||
**Created:**
|
||||
- `docs/LEGACY_CODE_INDEX.md` - Navigation guide
|
||||
- `docs/LEGACY_CODE_ANALYSIS.md` - Detailed technical analysis (286 lines)
|
||||
- `docs/LEGACY_CLEANUP_CHECKLIST.md` - Action-oriented procedures (239 lines)
|
||||
|
||||
**Impact:** 0 risk, all tests pass, no regressions
|
||||
|
||||
### Phase 3.2-3.4: Legacy Module Migration
|
||||
|
||||
**Commits:**
|
||||
- 1d244cf: Delete scroll.py (156 lines)
|
||||
- dfe42b0: Create engine/legacy/ subsystem and move render.py + layers.py
|
||||
- 526e5ae: Update production imports to engine.legacy.*
|
||||
- cda1358: Move legacy tests to tests/legacy/ directory
|
||||
|
||||
**Actions Taken:**
|
||||
|
||||
1. **Delete scroll.py (156 lines)**
|
||||
- Fully deprecated rendering orchestrator
|
||||
- No production code imports
|
||||
- Clean removal, 0 risk
|
||||
|
||||
2. **Create engine/legacy/ subsystem**
|
||||
- `engine/legacy/__init__.py` - Package documentation
|
||||
- `engine/legacy/render.py` - Moved from root (274 lines)
|
||||
- `engine/legacy/layers.py` - Moved from root (272 lines)
|
||||
|
||||
3. **Update Production Imports**
|
||||
- `engine/effects/__init__.py` - get_effect_chain() path
|
||||
- `engine/effects/controller.py` - Fallback import path
|
||||
- `engine/pipeline/adapters.py` - RenderStage & ItemsStage imports
|
||||
|
||||
4. **Move Legacy Tests**
|
||||
- `tests/legacy/test_render.py` - Moved from root
|
||||
- `tests/legacy/test_layers.py` - Moved from root
|
||||
- Updated all imports to use `engine.legacy.*`
|
||||
|
||||
**Impact:**
|
||||
- Core production code fully functional
|
||||
- Clear separation between legacy and modern code
|
||||
- All modern tests pass (67 new tests)
|
||||
- Ready for future removal of legacy modules
|
||||
|
||||
---
|
||||
|
||||
## Architecture Changes
|
||||
|
||||
### Before: Monolithic legacy code scattered throughout
|
||||
|
||||
```
|
||||
engine/
|
||||
├── emitters.py (unused)
|
||||
├── beautiful_mermaid.py (unused)
|
||||
├── animation.py (unused)
|
||||
├── pipeline_viz.py (unused)
|
||||
├── scroll.py (deprecated)
|
||||
├── render.py (legacy)
|
||||
├── layers.py (legacy)
|
||||
├── effects/
|
||||
│ └── controller.py (uses layers.py)
|
||||
└── pipeline/
|
||||
└── adapters.py (uses render.py + layers.py)
|
||||
|
||||
tests/
|
||||
├── test_render.py (tests legacy)
|
||||
├── test_layers.py (tests legacy)
|
||||
└── test_emitters.py (orphaned)
|
||||
```
|
||||
|
||||
### After: Clean separation of legacy and modern
|
||||
|
||||
```
|
||||
engine/
|
||||
├── legacy/
|
||||
│ ├── __init__.py
|
||||
│ ├── render.py (274 lines)
|
||||
│ └── layers.py (272 lines)
|
||||
├── effects/
|
||||
│ └── controller.py (imports engine.legacy.layers)
|
||||
└── pipeline/
|
||||
└── adapters.py (imports engine.legacy.*)
|
||||
|
||||
tests/
|
||||
├── test_data_sources.py (NEW - 19 tests)
|
||||
├── test_adapters.py (NEW - 37 tests)
|
||||
├── test_app.py (FIXED - 11 tests)
|
||||
└── legacy/
|
||||
├── test_render.py (moved, 24 passing tests)
|
||||
└── test_layers.py (moved, 30 passing tests)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test Statistics
|
||||
|
||||
### New Tests Added
|
||||
- `test_data_sources.py`: 19 tests (SourceItem, DataSources)
|
||||
- `test_adapters.py`: 37 tests (Pipeline stages)
|
||||
- `test_app.py`: 11 tests (fixed 7 failing tests)
|
||||
- **Total new:** 67 tests
|
||||
|
||||
### Test Categories
|
||||
- Unit tests: 67 new tests in core modules
|
||||
- Integration tests: 11 app tests covering pipeline orchestration
|
||||
- Legacy tests: 54 tests moved to `tests/legacy/` (6 pre-existing failures)
|
||||
|
||||
### Coverage Improvements
|
||||
| Module | Before | After | Improvement |
|
||||
|--------|--------|-------|-------------|
|
||||
| engine/app.py | 0-8% | 67% | +67% |
|
||||
| engine/data_sources/sources.py | 34% | 39% | +5% |
|
||||
| engine/pipeline/adapters.py | ~50% | 57% | +7% |
|
||||
| Overall | 35% | ~35% | (code cleanup offsets new tests) |
|
||||
|
||||
---
|
||||
|
||||
## Code Cleanup Statistics
|
||||
|
||||
### Phase 1-2: Dead Code Removal
|
||||
- **emitters.py:** 25 lines (0 references)
|
||||
- **beautiful_mermaid.py:** 4,107 lines (0 production usage)
|
||||
- **pipeline_viz.py:** 364 lines (0 production usage)
|
||||
- **animation.py:** 340 lines (0 imports)
|
||||
- **test_emitters.py:** 69 lines (orphaned)
|
||||
- **pipeline.py cleanup:** 25 lines (introspection methods)
|
||||
- **Total:** 4,930 lines removed, 0 risk
|
||||
|
||||
### Phase 3: Legacy Module Migration
|
||||
- **scroll.py:** 156 lines (deleted - fully deprecated)
|
||||
- **render.py:** 274 lines (moved to engine/legacy/)
|
||||
- **layers.py:** 272 lines (moved to engine/legacy/)
|
||||
- **Total moved:** 546 lines, properly organized
|
||||
|
||||
### Grand Total: 5,296 lines of dead/legacy code handled
|
||||
|
||||
---
|
||||
|
||||
## Git Commit History
|
||||
|
||||
```
|
||||
d14f850 refactor(remove): Remove RenderStage and ItemsStage from pipeline.py introspection (Phase 4.4)
|
||||
6fc3cbc refactor(remove): Delete RenderStage and ItemsStage classes (Phase 4.3)
|
||||
3e73ea0 refactor(remove-renderstage): Remove RenderStage usage from app.py (Phase 4.2)
|
||||
7c69086 refactor(deprecate): Add deprecation warning to RenderStage (Phase 4.1)
|
||||
0980279 docs: Add comprehensive session summary - Phase 2 & 3 complete
|
||||
cda1358 refactor(legacy): Move legacy tests to tests/legacy/ (Phase 3.4)
|
||||
526e5ae refactor(legacy): Update production imports to engine.legacy (Phase 3.3)
|
||||
dfe42b0 refactor(legacy): Create engine/legacy/ subsystem (Phase 3.2)
|
||||
1d244cf refactor(legacy): Delete scroll.py (Phase 3.1)
|
||||
0aa80f9 refactor(cleanup): Remove 340 lines of unused animation.py
|
||||
5762d5e refactor(cleanup): Remove 4,500 lines of dead code (Phase 1)
|
||||
28203ba test: Fix app.py integration tests - prevent pygame launch
|
||||
952b73c test: Add comprehensive pipeline adapter tests (37 tests)
|
||||
d9c7138 test: Add comprehensive data source tests (19 tests)
|
||||
c976b99 test(app): add focused integration tests for run_pipeline_mode
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quality Assurance
|
||||
|
||||
### Testing
|
||||
- ✅ All 67 new tests pass
|
||||
- ✅ All 11 app integration tests pass
|
||||
- ✅ 515 core tests passing (non-legacy)
|
||||
- ✅ No regressions in existing code
|
||||
- ✅ Legacy tests moved without breaking modern code
|
||||
|
||||
### Code Quality
|
||||
- ✅ All linting passes (ruff checks)
|
||||
- ✅ All syntax valid (Python 3.12 compatible)
|
||||
- ✅ Proper imports verified throughout codebase
|
||||
- ✅ Pre-commit hooks pass (format + lint)
|
||||
|
||||
### Documentation
|
||||
- ✅ 3 comprehensive legacy code analysis documents created
|
||||
- ✅ 4 phase migration strategy documented
|
||||
- ✅ Clear separation between legacy and modern code
|
||||
- ✅ Deprecation notices added to legacy modules
|
||||
|
||||
---
|
||||
|
||||
## Key Achievements
|
||||
|
||||
### Code Quality
|
||||
1. **Eliminated 5,296 lines of dead/legacy code** - cleaner codebase
|
||||
2. **Organized remaining legacy code** - `engine/legacy/` and `tests/legacy/`
|
||||
3. **Clear migration path** - legacy modules marked deprecated with timeline
|
||||
|
||||
### Testing Infrastructure
|
||||
1. **67 new comprehensive tests** - improved coverage of core modules
|
||||
2. **Fixed integration tests** - app.py tests now stable, prevent UI launch
|
||||
3. **Organized test structure** - legacy tests separated from modern tests
|
||||
|
||||
### Maintainability
|
||||
1. **Modern code fully functional** - 515 core tests passing
|
||||
2. **Legacy code isolated** - doesn't affect new pipeline architecture
|
||||
3. **Clear deprecation strategy** - timeline for removal documented
|
||||
|
||||
---
|
||||
|
||||
## Next Steps (Future Sessions)
|
||||
|
||||
### Completed
|
||||
- ✅ Document legacy code inventory - DONE (Phase 3)
|
||||
- ✅ Delete dead code - DONE (Phase 1-2)
|
||||
- ✅ Migrate legacy modules - DONE (Phase 3)
|
||||
- ✅ Remove deprecated adapters - DONE (Phase 4)
|
||||
|
||||
### Short Term (Phase 5)
|
||||
- Remove engine/legacy/ subsystem entirely
|
||||
- Delete tests/legacy/ directory
|
||||
- Clean up any remaining legacy imports in production code
|
||||
|
||||
### Long Term (Phase 6+)
|
||||
- Archive old rendering code to historical branch if needed
|
||||
- Final cleanup and code optimization
|
||||
- Performance profiling of modern pipeline
|
||||
|
||||
---
|
||||
|
||||
## Conclusion
|
||||
|
||||
This comprehensive 4-phase session successfully:
|
||||
|
||||
### Phase 2: Testing (67 new tests)
|
||||
1. ✅ Added comprehensive tests for DataSources, adapters, and app integration
|
||||
2. ✅ Improved coverage of core modules from ~35% → modern patterns
|
||||
3. ✅ Fixed integration tests to prevent UI launch in CI
|
||||
|
||||
### Phase 3: Legacy Organization (5,296 lines removed)
|
||||
1. ✅ Removed 4,930 lines of provably dead code
|
||||
2. ✅ Organized 546 lines of legacy code into dedicated subsystem
|
||||
3. ✅ Created clear separation: `engine/legacy/` and `tests/legacy/`
|
||||
|
||||
### Phase 4: Adapter Removal (280 lines removed)
|
||||
1. ✅ Deprecated RenderStage and ItemsStage
|
||||
2. ✅ Created ListDataSource replacement pattern
|
||||
3. ✅ Removed deprecated adapters and associated tests
|
||||
4. ✅ Updated pipeline introspection documentation
|
||||
|
||||
### Overall Results
|
||||
|
||||
**Code Quality:**
|
||||
- 5,576 total lines of legacy/dead code removed
|
||||
- Clean architecture with no deprecated patterns in use
|
||||
- Modern pipeline fully functional and testable
|
||||
|
||||
**Testing:**
|
||||
- 67 new tests added
|
||||
- 508 core tests passing (100% of modern code)
|
||||
- 19 tests skipped
|
||||
- 6 legacy test failures (pre-existing, unrelated to Phase 4)
|
||||
- Zero regressions in any phase
|
||||
|
||||
**Technical Debt:**
|
||||
- Reduced by 5,576 lines
|
||||
- Remaining legacy code (546 lines) isolated and marked for removal
|
||||
- Clear path to Phase 5: Complete removal of engine/legacy/
|
||||
|
||||
The codebase is now in excellent shape with:
|
||||
- ✅ No deprecated adapters in use
|
||||
- ✅ All modern code patterns adopted
|
||||
- ✅ Clear separation of concerns
|
||||
- ✅ Ready for next phase of cleanup
|
||||
|
||||
---
|
||||
|
||||
**End of Session Summary**
|
||||
Reference in New Issue
Block a user