klubhaus/sideline
2
0
Fork 0
forked from genewildish/Mainline
Code Issues 5 Pull Requests Projects 1 Releases Wiki Activity

feat: Add fast startup fetch and background caching

Browse Source 
- Add  for quick startup using first N feeds
- Add background thread for full fetch and caching
- Update  to use fast fetch
- Update docs and skills
This commit is contained in:
David Gwilliam
2026-03-19 22:38:55 -07:00
parent 4f2cf49a80
commit 7eaa441574
13 changed files with 393 additions and 220 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
.opencode/skills/mainline-architecture/SKILL.md
View File

@@ -29,17 +29,28 @@ class Stage(ABC):
return set() return set()
@property @property
def dependencies(self) -> list[str]: def dependencies(self) -> set[str]:
"""What this stage needs (e.g., ['source'])""" """What this stage needs (e.g., {'source'})"""
return [] return set()
``` ```
### Capability-Based Dependencies ### Capability-Based Dependencies
The Pipeline resolves dependencies using **prefix matching**: The Pipeline resolves dependencies using **prefix matching**:
- `"source"` matches `"source.headlines"`, `"source.poetry"`, etc. - `"source"` matches `"source.headlines"`, `"source.poetry"`, etc.
- `"camera.state"` matches the camera state capability
- This allows flexible composition without hardcoding specific stage names - This allows flexible composition without hardcoding specific stage names
### Minimum Capabilities
The pipeline requires these minimum capabilities to function:
- `"source"` - Data source capability
- `"render.output"` - Rendered content capability
- `"display.output"` - Display output capability
- `"camera.state"` - Camera state for viewport filtering
These are automatically injected if missing (auto-injection).
### DataType Enum ### DataType Enum
PureData-style data types for inlet/outlet validation: PureData-style data types for inlet/outlet validation:
@@ -76,3 +87,11 @@ Canvas tracks dirty regions automatically when content is written via `put_regio
- Use adapters (engine/pipeline/adapters.py) to wrap existing components as stages - Use adapters (engine/pipeline/adapters.py) to wrap existing components as stages
- Set `optional=True` for stages that can fail gracefully - Set `optional=True` for stages that can fail gracefully
- Use `stage_type` and `render_order` for execution ordering - Use `stage_type` and `render_order` for execution ordering
- Clock stages update state independently of data flow
## Sources
- engine/pipeline/core.py - Stage base class
- engine/pipeline/controller.py - Pipeline implementation
- engine/pipeline/adapters/ - Stage adapters
- docs/PIPELINE.md - Pipeline documentation

52
AGENTS.md
View File

@@ -267,15 +267,45 @@ The new Stage-based pipeline architecture provides capability-based dependency r
- **Stage** (`engine/pipeline/core.py`): Base class for pipeline stages - **Stage** (`engine/pipeline/core.py`): Base class for pipeline stages
- **Pipeline** (`engine/pipeline/controller.py`): Executes stages with capability-based dependency resolution - **Pipeline** (`engine/pipeline/controller.py`): Executes stages with capability-based dependency resolution
- **PipelineConfig** (`engine/pipeline/controller.py`): Configuration for pipeline instance
- **StageRegistry** (`engine/pipeline/registry.py`): Discovers and registers stages - **StageRegistry** (`engine/pipeline/registry.py`): Discovers and registers stages
- **Stage Adapters** (`engine/pipeline/adapters.py`): Wraps existing components as stages - **Stage Adapters** (`engine/pipeline/adapters.py`): Wraps existing components as stages
#### Pipeline Configuration
The `PipelineConfig` dataclass configures pipeline behavior:
```python
@dataclass
class PipelineConfig:
source: str = "headlines" # Data source identifier
display: str = "terminal" # Display backend identifier
camera: str = "vertical" # Camera mode identifier
effects: list[str] = field(default_factory=list) # List of effect names
enable_metrics: bool = True # Enable performance metrics
```
**Available sources**: `headlines`, `poetry`, `empty`, `list`, `image`, `metrics`, `cached`, `transform`, `composite`, `pipeline-inspect`
**Available displays**: `terminal`, `null`, `replay`, `websocket`, `pygame`, `moderngl`, `multi`
**Available camera modes**: `FEED`, `SCROLL`, `HORIZONTAL`, `OMNI`, `FLOATING`, `BOUNCE`, `RADIAL`
#### Capability-Based Dependencies #### Capability-Based Dependencies
Stages declare capabilities (what they provide) and dependencies (what they need). The Pipeline resolves dependencies using prefix matching: Stages declare capabilities (what they provide) and dependencies (what they need). The Pipeline resolves dependencies using prefix matching:
- `"source"` matches `"source.headlines"`, `"source.poetry"`, etc. - `"source"` matches `"source.headlines"`, `"source.poetry"`, etc.
- `"camera.state"` matches the camera state capability
- This allows flexible composition without hardcoding specific stage names - This allows flexible composition without hardcoding specific stage names
#### Minimum Capabilities
The pipeline requires these minimum capabilities to function:
- `"source"` - Data source capability
- `"render.output"` - Rendered content capability
- `"display.output"` - Display output capability
- `"camera.state"` - Camera state for viewport filtering
These are automatically injected if missing by the `ensure_minimum_capabilities()` method.
#### Sensor Framework #### Sensor Framework
- **Sensor** (`engine/sensors/__init__.py`): Base class for real-time input sensors - **Sensor** (`engine/sensors/__init__.py`): Base class for real-time input sensors
@@ -406,23 +436,23 @@ A skills library MCP server (`skills`) is available for capturing and tracking l
### Workflow ### Workflow
**Before starting work:** **Before starting work:**
1. Run `skills_list_skills` to see available skills 1. Run `local_skills_list_skills` to see available skills
2. Use `skills_peek_skill({name: "skill-name"})` to preview relevant skills 2. Use `local_skills_peek_skill({name: "skill-name"})` to preview relevant skills
3. Use `skills_skill_slice({name: "skill-name", query: "your question"})` to get relevant sections 3. Use `local_skills_skill_slice({name: "skill-name", query: "your question"})` to get relevant sections
**While working:** **While working:**
- If a skill was wrong or incomplete: `skills_update_skill` → `skills_record_assessment` → `skills_report_outcome({quality: 1})` - If a skill was wrong or incomplete: `local_skills_update_skill` → `local_skills_record_assessment` → `local_skills_report_outcome({quality: 1})`
- If a skill worked correctly: `skills_report_outcome({quality: 4})` (normal) or `quality: 5` (perfect) - If a skill worked correctly: `local_skills_report_outcome({quality: 4})` (normal) or `quality: 5` (perfect)
**End of session:** **End of session:**
- Run `skills_reflect_on_session({context_summary: "what you did"})` to identify new skills to capture - Run `local_skills_reflect_on_session({context_summary: "what you did"})` to identify new skills to capture
- Use `skills_create_skill` to add new skills - Use `local_skills_create_skill` to add new skills
- Use `skills_record_assessment` to score them - Use `local_skills_record_assessment` to score them
### Useful Tools ### Useful Tools
- `skills_review_stale_skills()` - Skills due for review (negative days_until_due) - `local_skills_review_stale_skills()` - Skills due for review (negative days_until_due)
- `skills_skills_report()` - Overview of entire collection - `local_skills_skills_report()` - Overview of entire collection
- `skills_validate_skill({name: "skill-name"})` - Load skill for review with sources - `local_skills_validate_skill({name: "skill-name"})` - Load skill for review with sources
### Agent Skills ### Agent Skills

234
docs/PIPELINE.md
View File

@@ -2,136 +2,160 @@
## Architecture Overview ## Architecture Overview
The Mainline pipeline uses a **Stage-based architecture** with **capability-based dependency resolution**. Stages declare capabilities (what they provide) and dependencies (what they need), and the Pipeline resolves dependencies using prefix matching.
``` ```
Sources (static/dynamic) → Fetch → Prepare → Scroll → Effects → Render → Display Source Stage → Render Stage → Effect Stages → Display Stage
↓ ↓
NtfyPoller ← MicMonitor (async) Camera Stage (provides camera.state capability)
``` ```
### Data Source Abstraction (sources_v2.py) ### Capability-Based Dependency Resolution
- **Static sources**: Data fetched once and cached (HeadlinesDataSource, PoetryDataSource) Stages declare capabilities and dependencies:
- **Dynamic sources**: Idempotent fetch for runtime updates (PipelineDataSource) - **Capabilities**: What the stage provides (e.g., `source`, `render.output`, `display.output`, `camera.state`)
- **SourceRegistry**: Discovery and management of data sources - **Dependencies**: What the stage needs (e.g., `source`, `render.output`, `camera.state`)
### Camera Modes The Pipeline resolves dependencies using **prefix matching**:
- `"source"` matches `"source.headlines"`, `"source.poetry"`, etc.
- `"camera.state"` matches the camera state capability provided by `CameraClockStage`
- This allows flexible composition without hardcoding specific stage names
- **Vertical**: Scroll up (default) ### Minimum Capabilities
- **Horizontal**: Scroll left
- **Omni**: Diagonal scroll
- **Floating**: Sinusoidal bobbing
- **Trace**: Follow network path node-by-node (for pipeline viz)
## Content to Display Rendering Pipeline The pipeline requires these minimum capabilities to function:
- `"source"` - Data source capability (provides raw items)
- `"render.output"` - Rendered content capability
- `"display.output"` - Display output capability
- `"camera.state"` - Camera state for viewport filtering
These are automatically injected if missing by the `ensure_minimum_capabilities()` method.
### Stage Registry
The `StageRegistry` discovers and registers stages automatically:
- Scans `engine/stages/` for stage implementations
- Registers stages by their declared capabilities
- Enables runtime stage discovery and composition
## Stage-Based Pipeline Flow
```mermaid ```mermaid
flowchart TD flowchart TD
subgraph Sources["Data Sources (v2)"] subgraph Stages["Stage Pipeline"]
Headlines[HeadlinesDataSource] subgraph SourceStage["Source Stage (provides: source.*)"]
Poetry[PoetryDataSource] Headlines[HeadlinesSource]
Pipeline[PipelineDataSource] Poetry[PoetrySource]
Registry[SourceRegistry] Pipeline[PipelineSource]
end end
subgraph SourcesLegacy["Data Sources (legacy)"] subgraph RenderStage["Render Stage (provides: render.*)"]
RSS[("RSS Feeds")] Render[RenderStage]
PoetryFeed[("Poetry Feed")] Canvas[Canvas]
Ntfy[("Ntfy Messages")] Camera[Camera]
Mic[("Microphone")] end
end
subgraph Fetch["Fetch Layer"] subgraph EffectStages["Effect Stages (provides: effect.*)"]
FC[fetch_all]
FP[fetch_poetry]
Cache[(Cache)]
end
subgraph Prepare["Prepare Layer"]
MB[make_block]
Strip[strip_tags]
Trans[translate]
end
subgraph Scroll["Scroll Engine"]
SC[StreamController]
CAM[Camera]
RTZ[render_ticker_zone]
Msg[render_message_overlay]
Grad[lr_gradient]
VT[vis_trunc / vis_offset]
end
subgraph Effects["Effect Pipeline"]
subgraph EffectsPlugins["Effect Plugins"]
Noise[NoiseEffect] Noise[NoiseEffect]
Fade[FadeEffect] Fade[FadeEffect]
Glitch[GlitchEffect] Glitch[GlitchEffect]
Firehose[FirehoseEffect] Firehose[FirehoseEffect]
Hud[HudEffect] Hud[HudEffect]
end end
EC[EffectChain]
ER[EffectRegistry] subgraph DisplayStage["Display Stage (provides: display.*)"]
Terminal[TerminalDisplay]
Pygame[PygameDisplay]
WebSocket[WebSocketDisplay]
Null[NullDisplay]
end
end end
subgraph Render["Render Layer"] subgraph Capabilities["Capability Map"]
BW[big_wrap] SourceCaps["source.headlines<br/>source.poetry<br/>source.pipeline"]
RL[render_line] RenderCaps["render.output<br/>render.canvas"]
EffectCaps["effect.noise<br/>effect.fade<br/>effect.glitch"]
DisplayCaps["display.output<br/>display.terminal"]
end end
subgraph Display["Display Backends"] SourceStage --> RenderStage
TD[TerminalDisplay] RenderStage --> EffectStages
PD[PygameDisplay] EffectStages --> DisplayStage
SD[SixelDisplay]
KD[KittyDisplay]
WSD[WebSocketDisplay]
ND[NullDisplay]
end
subgraph Async["Async Sources"] SourceStage --> SourceCaps
NTFY[NtfyPoller] RenderStage --> RenderCaps
MIC[MicMonitor] EffectStages --> EffectCaps
end DisplayStage --> DisplayCaps
subgraph Animation["Animation System"] style SourceStage fill:#f9f,stroke:#333
AC[AnimationController] style RenderStage fill:#bbf,stroke:#333
PR[Preset] style EffectStages fill:#fbf,stroke:#333
end style DisplayStage fill:#bfb,stroke:#333
Sources --> Fetch
RSS --> FC
PoetryFeed --> FP
FC --> Cache
FP --> Cache
Cache --> MB
Strip --> MB
Trans --> MB
MB --> SC
NTFY --> SC
SC --> RTZ
CAM --> RTZ
Grad --> RTZ
VT --> RTZ
RTZ --> EC
EC --> ER
ER --> EffectsPlugins
EffectsPlugins --> BW
BW --> RL
RL --> Display
Ntfy --> RL
Mic --> RL
MIC --> RL
style Sources fill:#f9f,stroke:#333
style Fetch fill:#bbf,stroke:#333
style Prepare fill:#bff,stroke:#333
style Scroll fill:#bfb,stroke:#333
style Effects fill:#fbf,stroke:#333
style Render fill:#ffb,stroke:#333
style Display fill:#bbf,stroke:#333
style Async fill:#fbb,stroke:#333
style Animation fill:#bfb,stroke:#333
``` ```
## Stage Adapters
Existing components are wrapped as Stages via adapters:
### Source Stage Adapter
- Wraps `HeadlinesDataSource`, `PoetryDataSource`, etc.
- Provides `source.*` capabilities
- Fetches data and outputs to pipeline buffer
### Render Stage Adapter
- Wraps `StreamController`, `Camera`, `render_ticker_zone`
- Provides `render.output` capability
- Processes content and renders to canvas
### Effect Stage Adapter
- Wraps `EffectChain` and individual effect plugins
- Provides `effect.*` capabilities
- Applies visual effects to rendered content
### Display Stage Adapter
- Wraps `TerminalDisplay`, `PygameDisplay`, etc.
- Provides `display.*` capabilities
- Outputs final buffer to display backend
## Pipeline Mutation API
The Pipeline supports dynamic mutation during runtime:
### Core Methods
- `add_stage(name, stage, initialize=True)` - Add a stage
- `remove_stage(name, cleanup=True)` - Remove a stage and rebuild execution order
- `replace_stage(name, new_stage, preserve_state=True)` - Replace a stage
- `swap_stages(name1, name2)` - Swap two stages
- `move_stage(name, after=None, before=None)` - Move a stage in execution order
- `enable_stage(name)` / `disable_stage(name)` - Enable/disable stages
### Safety Checks
- `can_hot_swap(name)` - Check if a stage can be safely hot-swapped
- `cleanup_stage(name)` - Clean up specific stage without removing it
### WebSocket Commands
The mutation API is accessible via WebSocket for remote control:
```json
{"action": "remove_stage", "stage": "stage_name"}
{"action": "swap_stages", "stage1": "name1", "stage2": "name2"}
{"action": "enable_stage", "stage": "stage_name"}
{"action": "cleanup_stage", "stage": "stage_name"}
```
## Camera Modes
The Camera supports the following modes:
- **FEED**: Single item view (static or rapid cycling)
- **SCROLL**: Smooth vertical scrolling (movie credits style)
- **HORIZONTAL**: Left/right movement
- **OMNI**: Combination of vertical and horizontal
- **FLOATING**: Sinusoidal/bobbing motion
- **BOUNCE**: DVD-style bouncing off edges
- **RADIAL**: Polar coordinate scanning (radar sweep)
Note: Camera state is provided by `CameraClockStage` (capability: `camera.state`) which updates independently of data flow. The `CameraStage` applies viewport transformations (capability: `camera`).
## Animation & Presets ## Animation & Presets
```mermaid ```mermaid
@@ -161,7 +185,7 @@ flowchart LR
Triggers --> Events Triggers --> Events
``` ```
## Camera Modes ## Camera Modes State Diagram
```mermaid ```mermaid
stateDiagram-v2 stateDiagram-v2

15
engine/app/main.py
View File

@@ -8,7 +8,7 @@ import time
from engine import config from engine import config
from engine.display import BorderMode, DisplayRegistry from engine.display import BorderMode, DisplayRegistry
from engine.effects import get_registry from engine.effects import get_registry
from engine.fetch import fetch_all, fetch_poetry, load_cache from engine.fetch import fetch_all, fetch_all_fast, fetch_poetry, load_cache, save_cache
from engine.pipeline import ( from engine.pipeline import (
Pipeline, Pipeline,
PipelineConfig, PipelineConfig,
@@ -208,7 +208,18 @@ def run_pipeline_mode_direct():
if cached: if cached:
source_items = cached source_items = cached
else: else:
source_items, _, _ = fetch_all() source_items = fetch_all_fast()
if source_items:
import threading
def background_fetch():
full_items, _, _ = fetch_all()
save_cache(full_items)
background_thread = threading.Thread(
target=background_fetch, daemon=True
)
background_thread.start()
elif source_name == "fixture": elif source_name == "fixture":
source_items = load_cache() source_items = load_cache()
if not source_items: if not source_items:

27
engine/app/pipeline_runner.py
View File

@@ -8,7 +8,7 @@ from typing import Any
from engine.display import BorderMode, DisplayRegistry from engine.display import BorderMode, DisplayRegistry
from engine.effects import get_registry from engine.effects import get_registry
from engine.fetch import fetch_all, fetch_poetry, load_cache from engine.fetch import fetch_all, fetch_all_fast, fetch_poetry, load_cache, save_cache
from engine.pipeline import Pipeline, PipelineConfig, PipelineContext, get_preset from engine.pipeline import Pipeline, PipelineConfig, PipelineContext, get_preset
from engine.pipeline.adapters import ( from engine.pipeline.adapters import (
EffectPluginStage, EffectPluginStage,
@@ -138,14 +138,7 @@ def run_pipeline_mode(preset_name: str = "demo"):
print("Error: Invalid viewport format. Use WxH (e.g., 40x15)") print("Error: Invalid viewport format. Use WxH (e.g., 40x15)")
sys.exit(1) sys.exit(1)
pipeline = Pipeline( pipeline = Pipeline(config=preset.to_config())
config=PipelineConfig(
source=preset.source,
display=preset.display,
camera=preset.camera,
effects=preset.effects,
)
)
print(" \033[38;5;245mFetching content...\033[0m") print(" \033[38;5;245mFetching content...\033[0m")
@@ -167,10 +160,24 @@ def run_pipeline_mode(preset_name: str = "demo"):
cached = load_cache() cached = load_cache()
if cached: if cached:
items = cached items = cached
print(f" \033[38;5;82mLoaded {len(items)} items from cache\033[0m")
elif preset.source == "poetry": elif preset.source == "poetry":
items, _, _ = fetch_poetry() items, _, _ = fetch_poetry()
else: else:
items, _, _ = fetch_all() items = fetch_all_fast()
if items:
print(
f" \033[38;5;82mFast start: {len(items)} items from first 5 sources\033[0m"
)
import threading
def background_fetch():
full_items, _, _ = fetch_all()
save_cache(full_items)
background_thread = threading.Thread(target=background_fetch, daemon=True)
background_thread.start()
if not items: if not items:
print(" \033[38;5;196mNo content available\033[0m") print(" \033[38;5;196mNo content available\033[0m")

14
engine/camera.py
View File

@@ -72,6 +72,17 @@ class Camera:
"""Shorthand for viewport_width.""" """Shorthand for viewport_width."""
return self.viewport_width return self.viewport_width
def set_speed(self, speed: float) -> None:
"""Set the camera scroll speed dynamically.
This allows camera speed to be modulated during runtime
via PipelineParams or directly.
Args:
speed: New speed value (0.0 = stopped, >0 = movement)
"""
self.speed = max(0.0, speed)
@property @property
def h(self) -> int: def h(self) -> int:
"""Shorthand for viewport_height.""" """Shorthand for viewport_height."""
@@ -373,10 +384,11 @@ class Camera:
truncated_line = vis_trunc(offset_line, viewport_width) truncated_line = vis_trunc(offset_line, viewport_width)
# Pad line to full viewport width to prevent ghosting when panning # Pad line to full viewport width to prevent ghosting when panning
# Skip padding for empty lines to preserve intentional blank lines
import re import re
visible_len = len(re.sub(r"\x1b\[[0-9;]*m", "", truncated_line)) visible_len = len(re.sub(r"\x1b\[[0-9;]*m", "", truncated_line))
if visible_len < viewport_width: if visible_len < viewport_width and visible_len > 0:
truncated_line += " " * (viewport_width - visible_len) truncated_line += " " * (viewport_width - visible_len)
horizontal_slice.append(truncated_line) horizontal_slice.append(truncated_line)

158
engine/fetch.py
View File

@@ -7,6 +7,7 @@ import json
import pathlib import pathlib
import re import re
import urllib.request import urllib.request
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime from datetime import datetime
from typing import Any from typing import Any
@@ -17,54 +18,98 @@ from engine.filter import skip, strip_tags
from engine.sources import FEEDS, POETRY_SOURCES from engine.sources import FEEDS, POETRY_SOURCES
from engine.terminal import boot_ln from engine.terminal import boot_ln
# Type alias for headline items
HeadlineTuple = tuple[str, str, str] HeadlineTuple = tuple[str, str, str]
DEFAULT_MAX_WORKERS = 10
FAST_START_SOURCES = 5
FAST_START_TIMEOUT = 3
# ─── SINGLE FEED ──────────────────────────────────────────
def fetch_feed(url: str) -> Any | None: def fetch_feed(url: str) -> tuple[str, Any] | tuple[None, None]:
"""Fetch and parse a single RSS feed URL.""" """Fetch and parse a single RSS feed URL. Returns (url, feed) tuple."""
try: try:
req = urllib.request.Request(url, headers={"User-Agent": "mainline/0.1"}) req = urllib.request.Request(url, headers={"User-Agent": "mainline/0.1"})
resp = urllib.request.urlopen(req, timeout=config.FEED_TIMEOUT) timeout = FAST_START_TIMEOUT if url in _fast_start_urls else config.FEED_TIMEOUT
return feedparser.parse(resp.read()) resp = urllib.request.urlopen(req, timeout=timeout)
return (url, feedparser.parse(resp.read()))
except Exception: except Exception:
return None return (url, None)
def _parse_feed(feed: Any, src: str) -> list[HeadlineTuple]:
"""Parse a feed and return list of headline tuples."""
items = []
if feed is None or (feed.bozo and not feed.entries):
return items
for e in feed.entries:
t = strip_tags(e.get("title", ""))
if not t or skip(t):
continue
pub = e.get("published_parsed") or e.get("updated_parsed")
try:
ts = datetime(*pub[:6]).strftime("%H:%M") if pub else "——:——"
except Exception:
ts = "——:——"
items.append((t, src, ts))
return items
def fetch_all_fast() -> list[HeadlineTuple]:
"""Fetch only the first N sources for fast startup."""
global _fast_start_urls
_fast_start_urls = set(list(FEEDS.values())[:FAST_START_SOURCES])
items: list[HeadlineTuple] = []
with ThreadPoolExecutor(max_workers=FAST_START_SOURCES) as executor:
futures = {
executor.submit(fetch_feed, url): src
for src, url in list(FEEDS.items())[:FAST_START_SOURCES]
}
for future in as_completed(futures):
src = futures[future]
url, feed = future.result()
if feed is None or (feed.bozo and not feed.entries):
boot_ln(src, "DARK", False)
continue
parsed = _parse_feed(feed, src)
if parsed:
items.extend(parsed)
boot_ln(src, f"LINKED [{len(parsed)}]", True)
else:
boot_ln(src, "EMPTY", False)
return items
# ─── ALL RSS FEEDS ────────────────────────────────────────
def fetch_all() -> tuple[list[HeadlineTuple], int, int]: def fetch_all() -> tuple[list[HeadlineTuple], int, int]:
"""Fetch all RSS feeds and return items, linked count, failed count.""" """Fetch all RSS feeds concurrently and return items, linked count, failed count."""
global _fast_start_urls
_fast_start_urls = set()
items: list[HeadlineTuple] = [] items: list[HeadlineTuple] = []
linked = failed = 0 linked = failed = 0
for src, url in FEEDS.items():
feed = fetch_feed(url) with ThreadPoolExecutor(max_workers=DEFAULT_MAX_WORKERS) as executor:
if feed is None or (feed.bozo and not feed.entries): futures = {executor.submit(fetch_feed, url): src for src, url in FEEDS.items()}
boot_ln(src, "DARK", False) for future in as_completed(futures):
failed += 1 src = futures[future]
continue url, feed = future.result()
n = 0 if feed is None or (feed.bozo and not feed.entries):
for e in feed.entries: boot_ln(src, "DARK", False)
t = strip_tags(e.get("title", "")) failed += 1
if not t or skip(t):
continue continue
pub = e.get("published_parsed") or e.get("updated_parsed") parsed = _parse_feed(feed, src)
try: if parsed:
ts = datetime(*pub[:6]).strftime("%H:%M") if pub else "——:——" items.extend(parsed)
except Exception: boot_ln(src, f"LINKED [{len(parsed)}]", True)
ts = "——:——" linked += 1
items.append((t, src, ts)) else:
n += 1 boot_ln(src, "EMPTY", False)
if n: failed += 1
boot_ln(src, f"LINKED [{n}]", True)
linked += 1
else:
boot_ln(src, "EMPTY", False)
failed += 1
return items, linked, failed return items, linked, failed
# ─── PROJECT GUTENBERG ────────────────────────────────────
def _fetch_gutenberg(url: str, label: str) -> list[HeadlineTuple]: def _fetch_gutenberg(url: str, label: str) -> list[HeadlineTuple]:
"""Download and parse stanzas/passages from a Project Gutenberg text.""" """Download and parse stanzas/passages from a Project Gutenberg text."""
try: try:
@@ -76,23 +121,21 @@ def _fetch_gutenberg(url: str, label: str) -> list[HeadlineTuple]:
.replace("\r\n", "\n") .replace("\r\n", "\n")
.replace("\r", "\n") .replace("\r", "\n")
) )
# Strip PG boilerplate
m = re.search(r"\*\*\*\s*START OF[^\n]*\n", text) m = re.search(r"\*\*\*\s*START OF[^\n]*\n", text)
if m: if m:
text = text[m.end() :] text = text[m.end() :]
m = re.search(r"\*\*\*\s*END OF", text) m = re.search(r"\*\*\*\s*END OF", text)
if m: if m:
text = text[: m.start()] text = text[: m.start()]
# Split on blank lines into stanzas/passages
blocks = re.split(r"\n{2,}", text.strip()) blocks = re.split(r"\n{2,}", text.strip())
items = [] items = []
for blk in blocks: for blk in blocks:
blk = " ".join(blk.split()) # flatten to one line blk = " ".join(blk.split())
if len(blk) < 20 or len(blk) > 280: if len(blk) < 20 or len(blk) > 280:
continue continue
if blk.isupper(): # skip all-caps headers if blk.isupper():
continue continue
if re.match(r"^[IVXLCDM]+\.?\s*$", blk): # roman numerals if re.match(r"^[IVXLCDM]+\.?\s*$", blk):
continue continue
items.append((blk, label, "")) items.append((blk, label, ""))
return items return items
@@ -100,29 +143,35 @@ def _fetch_gutenberg(url: str, label: str) -> list[HeadlineTuple]:
return [] return []
def fetch_poetry(): def fetch_poetry() -> tuple[list[HeadlineTuple], int, int]:
"""Fetch all poetry/literature sources.""" """Fetch all poetry/literature sources concurrently."""
items = [] items = []
linked = failed = 0 linked = failed = 0
for label, url in POETRY_SOURCES.items():
stanzas = _fetch_gutenberg(url, label) with ThreadPoolExecutor(max_workers=DEFAULT_MAX_WORKERS) as executor:
if stanzas: futures = {
boot_ln(label, f"LOADED [{len(stanzas)}]", True) executor.submit(_fetch_gutenberg, url, label): label
items.extend(stanzas) for label, url in POETRY_SOURCES.items()
linked += 1 }
else: for future in as_completed(futures):
boot_ln(label, "DARK", False) label = futures[future]
failed += 1 stanzas = future.result()
if stanzas:
boot_ln(label, f"LOADED [{len(stanzas)}]", True)
items.extend(stanzas)
linked += 1
else:
boot_ln(label, "DARK", False)
failed += 1
return items, linked, failed return items, linked, failed
# ─── CACHE ──────────────────────────────────────────────── _cache_dir = pathlib.Path(__file__).resolve().parent / "fixtures"
# Cache moved to engine/fixtures/headlines.json
_CACHE_DIR = pathlib.Path(__file__).resolve().parent / "fixtures"
def _cache_path(): def _cache_path():
return _CACHE_DIR / "headlines.json" return _cache_dir / "headlines.json"
def load_cache(): def load_cache():
@@ -144,3 +193,6 @@ def save_cache(items):
_cache_path().write_text(json.dumps({"items": items})) _cache_path().write_text(json.dumps({"items": items}))
except Exception: except Exception:
pass pass
_fast_start_urls: set = set()

20
engine/fixtures/headlines.json
View File

@@ -1,19 +1 @@
{ {"items": []}
"items": [
["Breaking: AI systems achieve breakthrough in natural language understanding", "TechDaily", "14:32"],
["Scientists discover new exoplanet in habitable zone", "ScienceNews", "13:15"],
["Global markets rally as inflation shows signs of cooling", "FinanceWire", "12:48"],
["New study reveals benefits of Mediterranean diet for cognitive health", "HealthJournal", "11:22"],
["Tech giants announce collaboration on AI safety standards", "TechDaily", "10:55"],
["Archaeologists uncover 3000-year-old city in desert", "HistoryNow", "09:30"],
["Renewable energy capacity surpasses fossil fuels for first time", "GreenWorld", "08:15"],
["Space agency prepares for next Mars mission launch window", "SpaceNews", "07:42"],
["New film breaks box office records on opening weekend", "EntertainmentHub", "06:18"],
["Local community raises funds for new library project", "CommunityPost", "05:30"],
["Quantum computing breakthrough could revolutionize cryptography", "TechWeekly", "15:20"],
["New species of deep-sea creature discovered in Pacific trench", "NatureToday", "14:05"],
["Electric vehicle sales surpass traditional cars in Europe", "AutoNews", "12:33"],
["Renowned artist unveils interactive AI-generated exhibition", "ArtsMonthly", "11:10"],
["Climate summit reaches historic agreement on emissions", "WorldNews", "09:55"]
]
}

10
engine/pipeline/adapters/camera.py
View File

@@ -62,6 +62,16 @@ class CameraClockStage(Stage):
if data is None: if data is None:
return data return data
# Update camera speed from params if explicitly set (for dynamic modulation)
# Only update if camera_speed in params differs from the default (1.0)
# This preserves camera speed set during construction
if (
ctx.params
and hasattr(ctx.params, "camera_speed")
and ctx.params.camera_speed != 1.0
):
self._camera.set_speed(ctx.params.camera_speed)
current_time = time.perf_counter() current_time = time.perf_counter()
dt = 0.0 dt = 0.0
if self._last_frame_time is not None: if self._last_frame_time is not None:

2
engine/pipeline/params.py
View File

@@ -32,7 +32,7 @@ class PipelineParams:
# Camera config # Camera config
camera_mode: str = "vertical" camera_mode: str = "vertical"
camera_speed: float = 1.0 camera_speed: float = 1.0 # Default speed
camera_x: int = 0 # For horizontal scrolling camera_x: int = 0 # For horizontal scrolling
# Effect config # Effect config

30
engine/pipeline/presets.py
View File

@@ -11,11 +11,14 @@ Loading order:
""" """
from dataclasses import dataclass, field from dataclasses import dataclass, field
from typing import Any from typing import TYPE_CHECKING, Any
from engine.display import BorderMode from engine.display import BorderMode
from engine.pipeline.params import PipelineParams from engine.pipeline.params import PipelineParams
if TYPE_CHECKING:
from engine.pipeline.controller import PipelineConfig
def _load_toml_presets() -> dict[str, Any]: def _load_toml_presets() -> dict[str, Any]:
"""Load presets from TOML file.""" """Load presets from TOML file."""
@@ -55,9 +58,10 @@ class PipelinePreset:
viewport_width: int = 80 # Viewport width in columns viewport_width: int = 80 # Viewport width in columns
viewport_height: int = 24 # Viewport height in rows viewport_height: int = 24 # Viewport height in rows
source_items: list[dict[str, Any]] | None = None # For ListDataSource source_items: list[dict[str, Any]] | None = None # For ListDataSource
enable_metrics: bool = True # Enable performance metrics collection
def to_params(self) -> PipelineParams: def to_params(self) -> PipelineParams:
"""Convert to PipelineParams.""" """Convert to PipelineParams (runtime configuration)."""
from engine.display import BorderMode from engine.display import BorderMode
params = PipelineParams() params = PipelineParams()
@@ -72,10 +76,27 @@ class PipelinePreset:
) )
params.camera_mode = self.camera params.camera_mode = self.camera
params.effect_order = self.effects.copy() params.effect_order = self.effects.copy()
# Note: camera_speed, viewport_width/height are not stored in PipelineParams params.camera_speed = self.camera_speed
# They are used directly from the preset object in pipeline_runner.py # Note: viewport_width/height are read from PipelinePreset directly
# in pipeline_runner.py, not from PipelineParams
return params return params
def to_config(self) -> "PipelineConfig":
"""Convert to PipelineConfig (static pipeline construction config).
PipelineConfig is used once at pipeline initialization and contains
the core settings that don't change during execution.
"""
from engine.pipeline.controller import PipelineConfig
return PipelineConfig(
source=self.source,
display=self.display,
camera=self.camera,
effects=self.effects.copy(),
enable_metrics=self.enable_metrics,
)
@classmethod @classmethod
def from_yaml(cls, name: str, data: dict[str, Any]) -> "PipelinePreset": def from_yaml(cls, name: str, data: dict[str, Any]) -> "PipelinePreset":
"""Create a PipelinePreset from YAML data.""" """Create a PipelinePreset from YAML data."""
@@ -91,6 +112,7 @@ class PipelinePreset:
viewport_width=data.get("viewport_width", 80), viewport_width=data.get("viewport_width", 80),
viewport_height=data.get("viewport_height", 24), viewport_height=data.get("viewport_height", 24),
source_items=data.get("source_items"), source_items=data.get("source_items"),
enable_metrics=data.get("enable_metrics", True),
) )

12
tests/test_fetch.py
View File

@@ -31,12 +31,12 @@ class TestFetchFeed:
@patch("engine.fetch.urllib.request.urlopen") @patch("engine.fetch.urllib.request.urlopen")
def test_fetch_network_error(self, mock_urlopen): def test_fetch_network_error(self, mock_urlopen):
"""Network error returns None.""" """Network error returns tuple with None feed."""
mock_urlopen.side_effect = Exception("Network error") mock_urlopen.side_effect = Exception("Network error")
result = fetch_feed("http://example.com/feed") url, feed = fetch_feed("http://example.com/feed")
assert result is None assert feed is None
class TestFetchAll: class TestFetchAll:
@@ -54,7 +54,7 @@ class TestFetchAll:
{"title": "Headline 1", "published_parsed": (2024, 1, 1, 12, 0, 0)}, {"title": "Headline 1", "published_parsed": (2024, 1, 1, 12, 0, 0)},
{"title": "Headline 2", "updated_parsed": (2024, 1, 2, 12, 0, 0)}, {"title": "Headline 2", "updated_parsed": (2024, 1, 2, 12, 0, 0)},
] ]
mock_fetch_feed.return_value = mock_feed mock_fetch_feed.return_value = ("http://example.com", mock_feed)
mock_skip.return_value = False mock_skip.return_value = False
mock_strip.side_effect = lambda x: x mock_strip.side_effect = lambda x: x
@@ -67,7 +67,7 @@ class TestFetchAll:
@patch("engine.fetch.boot_ln") @patch("engine.fetch.boot_ln")
def test_fetch_all_feed_error(self, mock_boot, mock_fetch_feed): def test_fetch_all_feed_error(self, mock_boot, mock_fetch_feed):
"""Feed error increments failed count.""" """Feed error increments failed count."""
mock_fetch_feed.return_value = None mock_fetch_feed.return_value = ("http://example.com", None)
items, linked, failed = fetch_all() items, linked, failed = fetch_all()
@@ -87,7 +87,7 @@ class TestFetchAll:
{"title": "Sports scores"}, {"title": "Sports scores"},
{"title": "Valid headline"}, {"title": "Valid headline"},
] ]
mock_fetch_feed.return_value = mock_feed mock_fetch_feed.return_value = ("http://example.com", mock_feed)
mock_skip.side_effect = lambda x: x == "Sports scores" mock_skip.side_effect = lambda x: x == "Sports scores"
mock_strip.side_effect = lambda x: x mock_strip.side_effect = lambda x: x

14
tests/test_pipeline_e2e.py
View File

@@ -218,9 +218,10 @@ class TestPipelineE2EHappyPath:
assert result.success assert result.success
frame = display.frames.get(timeout=1) frame = display.frames.get(timeout=1)
assert "Line A" in frame # Camera stage pads lines to viewport width, so check for substring match
assert "Line B" in frame assert any("Line A" in line for line in frame)
assert "Line C" in frame assert any("Line B" in line for line in frame)
assert any("Line C" in line for line in frame)
def test_empty_source_produces_empty_buffer(self): def test_empty_source_produces_empty_buffer(self):
"""An empty source should produce an empty (or blank) frame.""" """An empty source should produce an empty (or blank) frame."""
@@ -263,7 +264,10 @@ class TestPipelineE2EEffects:
assert result.success assert result.success
frame = display.frames.get(timeout=1) frame = display.frames.get(timeout=1)
assert "[FX1]" in frame, f"Marker not found in frame: {frame}" # Camera stage pads lines to viewport width, so check for substring match
assert any("[FX1]" in line for line in frame), (
f"Marker not found in frame: {frame}"
)
assert "Original" in "\n".join(frame) assert "Original" in "\n".join(frame)
def test_effect_chain_ordering(self): def test_effect_chain_ordering(self):
@@ -387,7 +391,7 @@ class TestPipelineE2EStageOrder:
# All regular (non-overlay) stages should have metrics # All regular (non-overlay) stages should have metrics
assert "source" in stage_names assert "source" in stage_names
assert "render" in stage_names assert "render" in stage_names
assert "display" in stage_names assert "queue" in stage_names # Display stage is named "queue" in the test
assert "effect_m" in stage_names assert "effect_m" in stage_names