klubhaus/sideline
2
0
Fork 0
forked from genewildish/Mainline
Code Issues 5 Pull Requests Projects 1 Releases Wiki Activity
Files
feature/graph-dsl
sideline/docs/hybrid-config.md
David Gwilliam 2c23c423a0 feat(hybrid): Add hybrid preset-graph configuration system 
Implement Option 5: Hybrid preset-graph system that combines preset
simplicity with graph flexibility, providing 70% reduction in config
file size compared to verbose node DSL.

## New Files

- engine/pipeline/hybrid_config.py - Core hybrid config parser
- examples/hybrid_config.toml - Example hybrid configuration (20 lines)
- examples/hybrid_visualization.py - Demo script using hybrid config
- tests/test_hybrid_config.py - Comprehensive test suite (17 tests)
- docs/hybrid-config.md - Complete documentation

## Key Features

1. **Concise Syntax** (70% smaller than verbose DSL):

2. **Automatic Connections**: Linear pipeline order is inferred

3. **Flexible Configuration**:
   - Inline objects:
   - Array notation:
   - Shorthand:

4. **Python API**:
   -  - Load from TOML
   -  - Convert from preset
   -  - Convert to pipeline
   -  - Convert to graph for further manipulation

## Usage

Loading hybrid configuration...
======================================================================
✓ Hybrid config loaded from hybrid_config.toml
  Source: headlines
  Camera: scroll
  Effects: 4
    - noise: intensity=0.3
    - fade: intensity=0.5
    - glitch: intensity=0.2
    - firehose: intensity=0.4
  Display: terminal
  Auto-injected stages for missing capabilities: ['camera_update', 'render']
✓ Pipeline created with 9 stages
  Stages: ['source', 'camera', 'noise', 'fade', 'glitch', 'firehose', 'display', 'camera_update', 'render']
[?25l✓ Pipeline initialized
Executing pipeline...
  > MIT Tech Review .............................. LINKED [10]
  > Quanta .............................. LINKED [5]
  > Phys.org .............................. LINKED [30]
  > Ars Technica .............................. LINKED [20]
  > Science Daily .............................. LINKED [60]
  > Nature .............................. LINKED [75]
  > New Scientist .............................. LINKED [99]
  > NASA .............................. LINKED [10]
  > BBC Business .............................. LINKED [54]
  > BBC Science .............................. LINKED [36]
  > MarketWatch .............................. LINKED [10]
  > NPR .............................. LINKED [10]
  > Economist .............................. LINKED [299]
  > Al Jazeera .............................. LINKED [25]
  > France24 .............................. LINKED [24]
  > Guardian World .............................. LINKED [45]
  > BBC World .............................. LINKED [28]
  > ABC Australia .............................. LINKED [23]
  > DW .............................. LINKED [124]
  > Smithsonian .............................. LINKED [10]
  > Aeon .............................. LINKED [20]
  > Wired .............................. LINKED [48]
  > The Hindu .............................. LINKED [60]
  > Japan Times .............................. LINKED [29]
  > Nautilus .............................. LINKED [10]
  > Guardian Culture .............................. LINKED [24]
  > Literary Hub .............................. LINKED [10]
  > The Conversation .............................. LINKED [48]
  > The Marginalian .............................. LINKED [20]
  > Longreads .............................. LINKED [25]
  > Der Spiegel .............................. LINKED [19]
  > Atlas Obscura .............................. LINKED [27]
  > SCMP ..............................The Download: OpenAI is building a fully automated researcher, and a psychedelic
 pe                e  r          o      in                     e  a
    -      n    b  an          t        l       r                i l
       nl ad     n    co  ut n      h  l h  a    h  t e  o  d d     t r   c e
C n  ua t m co    e s             a  h  a e p      s          o  f nd
     h  w r    o  n    ec  le  o e   cl  r  a  e
T e D w  o     h   en a o ’s new A     ns, and n x -  n  u   a  r  c   s
W  t do ne  nucl ar r   tors  ea  f   w s  ?
 h  Penta o   s  l nni g  or  I co p nies  o tr in    cl s   i d   t   def nse o
T    ownl  d   pe  I s  S mi  t    dea , an    ok’  CS M   ws it
T   J  lies T a   vol  d       er nt   y    K     i e
Qu nt m   y  o  ap   Pi  ee     n   r  g    rd
T e  a h T a  E p  i     y B     urve  Are  ver   er
Why     u a   d        Stil   t u  le W t  t      ll S uff?
W e e  ome  ee S     s, She S es   S ace T  e M  e o  F ac  l
      ｳ┋          ｳ ﾎ          ｳ ┆            ﾒ   ｷ          ｹ ┃            
Ligh -  s d     n  u      t s ar  f cia  str    r     a    mi   h  s   f    ng o
New resea  h exp  r s  h   a ad   of  i ms' u  q e t chnol g  s
L mi e  j    bl  k  oc     ob      o por  nit  s f   y  ng pe     in  oas  l  n
Are hu a    a ural   vi l nt?  ew re  arc  c  ll       o  - e   a s     ons
 a     m l      e e r  q a  s?
New      cove e  p o  s   ow  stro      eil   m t     a ter t   Ge in  8 e e
 o          a t                 g   3     a    g ye    b        r             b
How DICER cuts microRNAs with single-nucleotide precision                        LINKED [50]
======================================================================
Visualization Output:
======================================================================
The Download: OpenAI is building a fully automated researcher, and a psychedelic
 pe                e  r          o      in                     e  a
    -      n    b  an          t        l       r                i l
       nl ad     n    co  ut n      h  l h  a    h  t e  o  d d     t r   c e
C n  ua t m co    e s             a  h  a e p      s          o  f nd
     h  w r    o  n    ec  le  o e   cl  r  a  e
T e D w  o     h   en a o ’s new A     ns, and n x -  n  u   a  r  c   s
W  t do ne  nucl ar r   tors  ea  f   w s  ?
 h  Penta o   s  l nni g  or  I co p nies  o tr in    cl s   i d   t   def nse o
T    ownl  d   pe  I s  S mi  t    dea , an    ok’  CS M   ws it
T   J  lies T a   vol  d       er nt   y    K     i e
Qu nt m   y  o  ap   Pi  ee     n   r  g    rd
T e  a h T a  E p  i     y B     urve  Are  ver   er
Why     u a   d        Stil   t u  le W t  t      ll S uff?
W e e  ome  ee S     s, She S es   S ace T  e M  e o  F ac  l
      ｳ┋          ｳ ﾎ          ｳ ┆            ﾒ   ｷ          ｹ ┃            
Ligh -  s d     n  u      t s ar  f cia  str    r     a    mi   h  s   f    ng o
New resea  h exp  r s  h   a ad   of  i ms' u  q e t chnol g  s
L mi e  j    bl  k  oc     ob      o por  nit  s f   y  ng pe     in  oas  l  n
Are hu a    a ural   vi l nt?  ew re  arc  c  ll       o  - e   a s     ons
 a     m l      e e r  q a  s?
New      cove e  p o  s   ow  stro      eil   m t     a ter t   Ge in  8 e e
 o          a t                 g   3     a    g ye    b        r             b
How DICER cuts microRNAs with single-nucleotide precision
======================================================================
✓ Successfully rendered 24 lines

## Comparison

| Format | Lines | Use Case |
|--------|-------|----------|
| Preset | 10 | Simple configs |
| **Hybrid** | **20** | **Most use cases (recommended)** |
| Verbose DSL | 39 | Complex DAGs |

All existing functionality preserved - verbose node DSL still works.
2026-03-21 21:03:27 -07:00

5.7 KiB
Raw Permalink Blame History

Hybrid Preset-Graph Configuration

The hybrid configuration format combines the simplicity of presets with the flexibility of graphs, providing a concise way to define pipelines.

Overview

The hybrid format uses 70% less space than the verbose node-based DSL while providing the same functionality.

Comparison

Verbose Node DSL (39 lines):

[nodes.source]
type = "source"
source = "headlines"

[nodes.camera]
type = "camera"
mode = "scroll"
speed = 1.0

[nodes.noise]
type = "effect"
effect = "noise"
intensity = 0.3

[nodes.display]
type = "display"
backend = "terminal"

[connections]
list = ["source -> camera -> noise -> display"]

Hybrid Config (20 lines):

[pipeline]
source = "headlines"

camera = { mode = "scroll", speed = 1.0 }

effects = [
    { name = "noise", intensity = 0.3 }
]

display = { backend = "terminal" }

Syntax

Basic Structure

[pipeline]
source = "headlines"
camera = { mode = "scroll", speed = 1.0 }
effects = [
    { name = "noise", intensity = 0.3 },
    { name = "fade", intensity = 0.5 }
]
display = { backend = "terminal", positioning = "mixed" }

Configuration Options

Source

source = "headlines"  # Built-in source: headlines, poetry, empty, etc.

Camera

# Inline object notation
camera = { mode = "scroll", speed = 1.0 }

# Or shorthand (uses defaults)
camera = "scroll"

Available modes: scroll, feed, horizontal, omni, floating, bounce, radial

Effects

# Array of effect configurations
effects = [
    { name = "noise", intensity = 0.3 },
    { name = "fade", intensity = 0.5, enabled = true }
]

# Or shorthand (uses defaults)
effects = ["noise", "fade"]

Available effects: noise, fade, glitch, firehose, tint, hud, etc.

Display

# Inline object notation
display = { backend = "terminal", positioning = "mixed" }

# Or shorthand
display = "terminal"

Available backends: terminal, null, websocket, pygame

Viewport Settings

[pipeline]
viewport_width = 80
viewport_height = 24

Usage Examples

Minimal Configuration

[pipeline]
source = "headlines"
display = "terminal"

With Camera and Effects

[pipeline]
source = "headlines"
camera = { mode = "scroll", speed = 1.0 }
effects = [
    { name = "noise", intensity = 0.3 },
    { name = "fade", intensity = 0.5 }
]
display = { backend = "terminal", positioning = "mixed" }

Full Configuration

[pipeline]
source = "poetry"
camera = { mode = "scroll", speed = 1.5 }
effects = [
    { name = "noise", intensity = 0.2 },
    { name = "fade", intensity = 0.4 },
    { name = "glitch", intensity = 0.3 },
    { name = "firehose", intensity = 0.5 }
]
display = { backend = "terminal", positioning = "mixed" }
viewport_width = 100
viewport_height = 30

Python API

Loading from TOML File

from engine.pipeline.hybrid_config import load_hybrid_config

config = load_hybrid_config("examples/hybrid_config.toml")
pipeline = config.to_pipeline()

Creating Config Programmatically

from engine.pipeline.hybrid_config import (
    PipelineConfig,
    CameraConfig,
    EffectConfig,
    DisplayConfig,
)

config = PipelineConfig(
    source="headlines",
    camera=CameraConfig(mode="scroll", speed=1.0),
    effects=[
        EffectConfig(name="noise", intensity=0.3),
        EffectConfig(name="fade", intensity=0.5),
    ],
    display=DisplayConfig(backend="terminal", positioning="mixed"),
)

pipeline = config.to_pipeline(viewport_width=80, viewport_height=24)

Converting to Graph

from engine.pipeline.hybrid_config import PipelineConfig

config = PipelineConfig(source="headlines", display={"backend": "terminal"})
graph = config.to_graph()  # Returns Graph object for further manipulation

How It Works

The hybrid config system:

  1. Parses TOML into a PipelineConfig dataclass
  2. Converts to Graph internally using automatic linear connections
  3. Reuses existing adapter to convert graph to pipeline stages
  4. Maintains backward compatibility with verbose node DSL

Automatic Connection Logic

The system automatically creates linear connections:

source -> camera -> effects[0] -> effects[1] -> ... -> display

This covers 90% of use cases. For complex DAGs, use the verbose node DSL.

Migration Guide

From Presets

The hybrid format is very similar to presets:

Preset:

[presets.custom]
source = "headlines"
effects = ["noise", "fade"]
display = "terminal"

Hybrid:

[pipeline]
source = "headlines"
effects = ["noise", "fade"]
display = "terminal"

The main difference is using [pipeline] instead of [presets.custom].

From Verbose Node DSL

Old (39 lines):

[nodes.source] type = "source" source = "headlines"
[nodes.camera] type = "camera" mode = "scroll"
[nodes.noise] type = "effect" effect = "noise" intensity = 0.3
[nodes.display] type = "display" backend = "terminal"
[connections] list = ["source -> camera -> noise -> display"]

New (14 lines):

[pipeline]
source = "headlines"
camera = { mode = "scroll" }
effects = [{ name = "noise", intensity = 0.3 }]
display = { backend = "terminal" }

When to Use Each Format

Format Use When Lines (example)
Preset Simple configurations, no effect intensity tuning 10
Hybrid Most common use cases, need intensity tuning 20
Verbose Node DSL Complex DAGs, branching, custom connections 39
Python API Dynamic configuration, programmatic generation N/A

Examples

See examples/hybrid_config.toml for a complete working example.

Run the demo:

python examples/hybrid_visualization.py
Reference in New Issue View Git Blame Copy Permalink