Mainline/docs/PIPELINE.md

Mainline Pipeline

Architecture Overview

The Mainline pipeline uses a Stage-based architecture with capability-based dependency resolution. Stages declare capabilities (what they provide) and dependencies (what they need), and the Pipeline resolves dependencies using prefix matching.

Source Stage → Render Stage → Effect Stages → Display Stage
      ↓
Camera Stage (provides camera.state capability)

Capability-Based Dependency Resolution

Stages declare capabilities and dependencies:

• Capabilities: What the stage provides (e.g., source, render.output, display.output, camera.state)
• Dependencies: What the stage needs (e.g., source, render.output, camera.state)

The Pipeline resolves dependencies using prefix matching:

• "source" matches "source.headlines", "source.poetry", etc.
• "camera.state" matches the camera state capability provided by CameraClockStage
• This allows flexible composition without hardcoding specific stage names

Minimum Capabilities

The pipeline requires these minimum capabilities to function:

• "source" - Data source capability (provides raw items)
• "render.output" - Rendered content capability
• "display.output" - Display output capability
• "camera.state" - Camera state for viewport filtering

These are automatically injected if missing by the ensure_minimum_capabilities() method.

Stage Registry

The StageRegistry discovers and registers stages automatically:

• Scans engine/stages/ for stage implementations
• Registers stages by their declared capabilities
• Enables runtime stage discovery and composition

Stage-Based Pipeline Flow

flowchart TD
    subgraph Stages["Stage Pipeline"]
        subgraph SourceStage["Source Stage (provides: source.*)"]
            Headlines[HeadlinesSource]
            Poetry[PoetrySource]
            Pipeline[PipelineSource]
        end

        subgraph RenderStage["Render Stage (provides: render.*)"]
            Render[RenderStage]
            Canvas[Canvas]
            Camera[Camera]
        end

        subgraph EffectStages["Effect Stages (provides: effect.*)"]
            Noise[NoiseEffect]
            Fade[FadeEffect]
            Glitch[GlitchEffect]
            Firehose[FirehoseEffect]
            Hud[HudEffect]
        end

        subgraph DisplayStage["Display Stage (provides: display.*)"]
            Terminal[TerminalDisplay]
            Pygame[PygameDisplay]
            WebSocket[WebSocketDisplay]
            Null[NullDisplay]
        end
    end

    subgraph Capabilities["Capability Map"]
        SourceCaps["source.headlines<br/>source.poetry<br/>source.pipeline"]
        RenderCaps["render.output<br/>render.canvas"]
        EffectCaps["effect.noise<br/>effect.fade<br/>effect.glitch"]
        DisplayCaps["display.output<br/>display.terminal"]
    end

    SourceStage --> RenderStage
    RenderStage --> EffectStages
    EffectStages --> DisplayStage

    SourceStage --> SourceCaps
    RenderStage --> RenderCaps
    EffectStages --> EffectCaps
    DisplayStage --> DisplayCaps

    style SourceStage fill:#f9f,stroke:#333
    style RenderStage fill:#bbf,stroke:#333
    style EffectStages fill:#fbf,stroke:#333
    style DisplayStage fill:#bfb,stroke:#333

Stage Adapters

Existing components are wrapped as Stages via adapters:

Source Stage Adapter

• Wraps HeadlinesDataSource, PoetryDataSource, etc.
• Provides source.* capabilities
• Fetches data and outputs to pipeline buffer

Render Stage Adapter

• Wraps StreamController, Camera, render_ticker_zone
• Provides render.output capability
• Processes content and renders to canvas

Effect Stage Adapter

• Wraps EffectChain and individual effect plugins
• Provides effect.* capabilities
• Applies visual effects to rendered content

Display Stage Adapter

• Wraps TerminalDisplay, PygameDisplay, etc.
• Provides display.* capabilities
• Outputs final buffer to display backend

Pipeline Mutation API

The Pipeline supports dynamic mutation during runtime:

Core Methods

• add_stage(name, stage, initialize=True) - Add a stage
• remove_stage(name, cleanup=True) - Remove a stage and rebuild execution order
• replace_stage(name, new_stage, preserve_state=True) - Replace a stage
• swap_stages(name1, name2) - Swap two stages
• move_stage(name, after=None, before=None) - Move a stage in execution order
• enable_stage(name) / disable_stage(name) - Enable/disable stages

Safety Checks

• can_hot_swap(name) - Check if a stage can be safely hot-swapped
• cleanup_stage(name) - Clean up specific stage without removing it

WebSocket Commands

The mutation API is accessible via WebSocket for remote control:

{"action": "remove_stage", "stage": "stage_name"}
{"action": "swap_stages", "stage1": "name1", "stage2": "name2"}
{"action": "enable_stage", "stage": "stage_name"}
{"action": "cleanup_stage", "stage": "stage_name"}

Camera Modes

The Camera supports the following modes:

• FEED: Single item view (static or rapid cycling)
• SCROLL: Smooth vertical scrolling (movie credits style)
• HORIZONTAL: Left/right movement
• OMNI: Combination of vertical and horizontal
• FLOATING: Sinusoidal/bobbing motion
• BOUNCE: DVD-style bouncing off edges
• RADIAL: Polar coordinate scanning (radar sweep)

Note: Camera state is provided by CameraClockStage (capability: camera.state) which updates independently of data flow. The CameraStage applies viewport transformations (capability: camera).

Animation & Presets

flowchart LR
    subgraph Preset["Preset"]
        PP[PipelineParams]
        AC[AnimationController]
    end
    
    subgraph AnimationController["AnimationController"]
        Clock[Clock]
        Events[Events]
        Triggers[Triggers]
    end
    
    subgraph Triggers["Trigger Types"]
        TIME[TIME]
        FRAME[FRAME]
        CYCLE[CYCLE]
        COND[CONDITION]
        MANUAL[MANUAL]
    end
    
    PP --> AC
    Clock --> AC
    Events --> AC
    Triggers --> Events

Camera Modes State Diagram

stateDiagram-v2
    [*] --> Vertical
    Vertical --> Horizontal: mode change
    Horizontal --> Omni: mode change
    Omni --> Floating: mode change
    Floating --> Trace: mode change
    Trace --> Vertical: mode change

    state Vertical {
        [*] --> ScrollUp
        ScrollUp --> ScrollUp: +y each frame
    }

    state Horizontal {
        [*] --> ScrollLeft
        ScrollLeft --> ScrollLeft: +x each frame
    }

    state Omni {
        [*] --> Diagonal
        Diagonal --> Diagonal: +x, +y each frame
    }

    state Floating {
        [*] --> Bobbing
        Bobbing --> Bobbing: sin(time) for x,y
    }

    state Trace {
        [*] --> FollowPath
        FollowPath --> FollowPath: node by node
    }