forked from genewildish/Mainline
David Gwilliam
915598629a
Add comprehensive documentation for the graph-based pipeline DSL: - docs/graph-dsl.md: Complete DSL reference with TOML, Python, and CLI syntax - docs/GRAPH_SYSTEM_SUMMARY.md: Implementation overview and architecture - examples/graph_dsl_demo.py: Demonstrates imperative Python API usage - examples/test_graph_integration.py: Integration test verifying pipeline execution The documentation follows a wiki-like approach with navigable structure: - Overview section explaining the concept - Syntax examples for each format (TOML, Python, CLI) - Node type reference table - Advanced features section - Comparison with old XYZStage approach This provides users with multiple entry points to understand and use the new graph-based pipeline system.
211 lines
4.5 KiB
Markdown
211 lines
4.5 KiB
Markdown
# Graph-Based Pipeline DSL
|
|
|
|
This document describes the new graph-based DSL for defining pipelines in Mainline.
|
|
|
|
## Overview
|
|
|
|
The graph DSL represents pipelines as nodes and connections, replacing the verbose `XYZStage` naming convention with a more intuitive graph abstraction.
|
|
|
|
## TOML Syntax
|
|
|
|
### Basic Pipeline
|
|
|
|
```toml
|
|
[nodes.source]
|
|
type = "source"
|
|
source = "headlines"
|
|
|
|
[nodes.camera]
|
|
type = "camera"
|
|
mode = "scroll"
|
|
|
|
[nodes.display]
|
|
type = "display"
|
|
backend = "terminal"
|
|
|
|
[connections]
|
|
list = ["source -> camera -> display"]
|
|
```
|
|
|
|
### With Effects
|
|
|
|
```toml
|
|
[nodes.source]
|
|
type = "source"
|
|
source = "headlines"
|
|
|
|
[nodes.noise]
|
|
type = "effect"
|
|
effect = "noise"
|
|
intensity = 0.5
|
|
|
|
[nodes.fade]
|
|
type = "effect"
|
|
effect = "fade"
|
|
intensity = 0.8
|
|
|
|
[nodes.display]
|
|
type = "display"
|
|
backend = "terminal"
|
|
|
|
[connections]
|
|
list = ["source -> noise -> fade -> display"]
|
|
```
|
|
|
|
### With Positioning
|
|
|
|
```toml
|
|
[nodes.source]
|
|
type = "source"
|
|
source = "headlines"
|
|
|
|
[nodes.position]
|
|
type = "position"
|
|
mode = "mixed"
|
|
|
|
[nodes.display]
|
|
type = "display"
|
|
backend = "terminal"
|
|
|
|
[connections]
|
|
list = ["source -> position -> display"]
|
|
```
|
|
|
|
## Python API
|
|
|
|
### Basic Construction
|
|
|
|
```python
|
|
from engine.pipeline.graph import Graph, NodeType
|
|
|
|
graph = Graph()
|
|
graph.node("source", NodeType.SOURCE, source="headlines")
|
|
graph.node("camera", NodeType.CAMERA, mode="scroll")
|
|
graph.node("display", NodeType.DISPLAY, backend="terminal")
|
|
graph.chain("source", "camera", "display")
|
|
|
|
pipeline = graph_to_pipeline(graph)
|
|
```
|
|
|
|
### With Effects
|
|
|
|
```python
|
|
from engine.pipeline.graph import Graph, NodeType
|
|
|
|
graph = Graph()
|
|
graph.node("source", NodeType.SOURCE, source="headlines")
|
|
graph.node("noise", NodeType.EFFECT, effect="noise", intensity=0.5)
|
|
graph.node("fade", NodeType.EFFECT, effect="fade", intensity=0.8)
|
|
graph.node("display", NodeType.DISPLAY, backend="terminal")
|
|
graph.chain("source", "noise", "fade", "display")
|
|
|
|
pipeline = graph_to_pipeline(graph)
|
|
```
|
|
|
|
### Dictionary/JSON Input
|
|
|
|
```python
|
|
from engine.pipeline.graph_adapter import dict_to_pipeline
|
|
|
|
data = {
|
|
"nodes": {
|
|
"source": "headlines",
|
|
"noise": {"type": "effect", "effect": "noise", "intensity": 0.5},
|
|
"display": {"type": "display", "backend": "terminal"}
|
|
},
|
|
"connections": ["source -> noise -> display"]
|
|
}
|
|
|
|
pipeline = dict_to_pipeline(data)
|
|
```
|
|
|
|
## CLI Usage
|
|
|
|
### Using Graph Config File
|
|
|
|
```bash
|
|
mainline --graph-config pipeline.toml
|
|
```
|
|
|
|
### Inline Graph Definition
|
|
|
|
```bash
|
|
mainline --graph 'source:headlines -> noise:noise:0.5 -> display:terminal'
|
|
```
|
|
|
|
### With Preset Override
|
|
|
|
```bash
|
|
mainline --preset demo --graph-modify 'add:noise:0.5 after:source'
|
|
```
|
|
|
|
## Node Types
|
|
|
|
| Type | Description | Config Options |
|
|
|------|-------------|----------------|
|
|
| `source` | Data source | `source`: "headlines", "poetry", "empty", etc. |
|
|
| `camera` | Viewport camera | `mode`: "scroll", "feed", "horizontal", etc. `speed`: float |
|
|
| `effect` | Visual effect | `effect`: effect name, `intensity`: 0.0-1.0 |
|
|
| `position` | Positioning mode | `mode`: "absolute", "relative", "mixed" |
|
|
| `display` | Output backend | `backend`: "terminal", "null", "websocket", etc. |
|
|
| `render` | Text rendering | (auto-injected) |
|
|
| `overlay` | Message overlay | (auto-injected) |
|
|
|
|
## Advanced Features
|
|
|
|
### Conditional Connections
|
|
|
|
```toml
|
|
[connections]
|
|
list = ["source -> camera -> display"]
|
|
# Effects can be conditionally enabled/disabled
|
|
```
|
|
|
|
### Parameter Binding
|
|
|
|
```toml
|
|
[nodes.noise]
|
|
type = "effect"
|
|
effect = "noise"
|
|
intensity = 1.0
|
|
# intensity can be bound to sensor values at runtime
|
|
```
|
|
|
|
### Pipeline Inspection
|
|
|
|
```toml
|
|
[nodes.inspect]
|
|
type = "pipeline-inspect"
|
|
# Renders live pipeline visualization
|
|
```
|
|
|
|
## Comparison with Stage-Based Approach
|
|
|
|
### Old (Stage-Based)
|
|
|
|
```python
|
|
pipeline = Pipeline()
|
|
pipeline.add_stage("source", DataSourceStage(HeadlinesDataSource()))
|
|
pipeline.add_stage("camera", CameraStage(Camera.scroll()))
|
|
pipeline.add_stage("render", FontStage())
|
|
pipeline.add_stage("noise", EffectPluginStage(noise_effect))
|
|
pipeline.add_stage("display", DisplayStage(terminal_display))
|
|
```
|
|
|
|
### New (Graph-Based)
|
|
|
|
```python
|
|
graph = Graph()
|
|
graph.node("source", NodeType.SOURCE, source="headlines")
|
|
graph.node("camera", NodeType.CAMERA, mode="scroll")
|
|
graph.node("noise", NodeType.EFFECT, effect="noise")
|
|
graph.node("display", NodeType.DISPLAY, backend="terminal")
|
|
graph.chain("source", "camera", "noise", "display")
|
|
pipeline = graph_to_pipeline(graph)
|
|
```
|
|
|
|
The graph system automatically:
|
|
- Inserts the render stage between camera and effects
|
|
- Handles capability-based dependency resolution
|
|
- Auto-injects required stages (camera_update, render, etc.)
|