klubhaus/sideline
2
0
Fork 0
forked from genewildish/Mainline
Code Issues 5 Pull Requests Projects 1 Releases Wiki Activity

Add Streaming display backend #26

New Issue
Closed
opened 2026-03-18 10:02:51 +00:00 by david · 2 comments
david commented 2026-03-18 10:02:51 +00:00
Owner
Copy Link

Summary

Add a new "Streaming" display backend that renders to streaming formats (HLS, DASH, or raw frames via WebSocket).

Motivation

Currently Mainline supports:

  • Terminal (ANSI)
  • Pygame (windowed)
  • WebSocket (browser)
  • Sixel (graphics terminals)
  • Null (testing)

A "Streaming" display would enable:

  • Live streaming to Twitch/OBS via RTMP
  • Recording to file (MP4/WebM)
  • Multi-user remote viewing

Potential Approaches

  1. RTMP output - Stream directly to RTMP server (OBS can ingest)
  2. WebM segmenter - Write WebM fragments for HLS/DASH
  3. Frame callback - Provide callback for custom render targets

Context

See existing display implementations in engine/display/backends/

Additional context

From our architecture docs, the Display protocol requires:

  • init(width, height, reuse=False)
  • show(buffer: list[str], border: bool = False)
  • clear() -> None
  • cleanup() -> None

Optional methods:

  • is_quit_requested() -> bool
  • clear_quit_request() -> None
## Summary Add a new "Streaming" display backend that renders to streaming formats (HLS, DASH, or raw frames via WebSocket). ## Motivation Currently Mainline supports: - Terminal (ANSI) - Pygame (windowed) - WebSocket (browser) - Sixel (graphics terminals) - Null (testing) A "Streaming" display would enable: - Live streaming to Twitch/OBS via RTMP - Recording to file (MP4/WebM) - Multi-user remote viewing ## Potential Approaches 1. **RTMP output** - Stream directly to RTMP server (OBS can ingest) 2. **WebM segmenter** - Write WebM fragments for HLS/DASH 3. **Frame callback** - Provide callback for custom render targets ## Context See existing display implementations in `engine/display/backends/` ## Additional context From our architecture docs, the Display protocol requires: - `init(width, height, reuse=False)` - `show(buffer: list[str], border: bool = False)` - `clear() -> None` - `cleanup() -> None` Optional methods: - `is_quit_requested() -> bool` - `clear_quit_request() -> None`
david commented 2026-03-19 03:41:34 +00:00
Author
Owner
Copy Link

Clarification (2026-03-18)

After analyzing the codebase, there's some confusion about the scope of this issue. Let me clarify:

Current State

The existing WebSocketDisplay in engine/display/backends/websocket.py already streams frames to web clients. However, there's a TODO comment indicating it needs optimization:

TODO: Transform to a true streaming backend with:
- Proper WebSocket message streaming (currently sends full buffer each frame)
- Connection pooling and backpressure handling
- Binary protocol for efficiency (instead of JSON)
- Client management with proper async handling

Decision: Focus on WebSocket Optimization

Rather than implementing a new "Streaming" display backend (RTMP/HLS), we should optimize the existing WebSocket display. This will:

  1. Reduce bandwidth - Binary protocol instead of JSON
  2. Lower latency - Differential updates instead of full frames
  3. Improve scalability - Better connection handling

Implementation Plan (Option B - Binary Protocol with Compression)

  1. Design binary protocol:

    Message format:
- Type: 1 byte (frame, diff, state, clear)
- Width: 2 bytes (uint16)
- Height: 2 bytes (uint16)
- Data: compressed (gzip or simple compression)

  2. Compression approach:

    • Run-length encoding for repeated lines
    • Delta compression for changed regions
    • Periodic full frame sync for recovery

  3. Backward compatibility:

    • Keep JSON fallback for older clients
    • Client announces capability during connection

Dependencies

This optimization enables Issue #33 (Web Editor) by providing efficient real-time updates.

Alternative: RTMP Streaming

If RTMP streaming is still needed, that could be a separate issue. The current scope is just WebSocket optimization.

Should we proceed with this plan?

## Clarification (2026-03-18) After analyzing the codebase, there's some confusion about the scope of this issue. Let me clarify: ### Current State The existing `WebSocketDisplay` in `engine/display/backends/websocket.py` already streams frames to web clients. However, there's a TODO comment indicating it needs optimization: ``` TODO: Transform to a true streaming backend with: - Proper WebSocket message streaming (currently sends full buffer each frame) - Connection pooling and backpressure handling - Binary protocol for efficiency (instead of JSON) - Client management with proper async handling ``` ### Decision: Focus on WebSocket Optimization Rather than implementing a new "Streaming" display backend (RTMP/HLS), we should optimize the existing WebSocket display. This will: 1. **Reduce bandwidth** - Binary protocol instead of JSON 2. **Lower latency** - Differential updates instead of full frames 3. **Improve scalability** - Better connection handling ### Implementation Plan (Option B - Binary Protocol with Compression) 1. **Design binary protocol:** ``` Message format: - Type: 1 byte (frame, diff, state, clear) - Width: 2 bytes (uint16) - Height: 2 bytes (uint16) - Data: compressed (gzip or simple compression) ``` 2. **Compression approach:** - Run-length encoding for repeated lines - Delta compression for changed regions - Periodic full frame sync for recovery 3. **Backward compatibility:** - Keep JSON fallback for older clients - Client announces capability during connection ### Dependencies This optimization enables Issue #33 (Web Editor) by providing efficient real-time updates. ### Alternative: RTMP Streaming If RTMP streaming is still needed, that could be a separate issue. The current scope is just WebSocket optimization. Should we proceed with this plan?
david commented 2026-03-19 05:34:52 +00:00
Author
Owner
Copy Link

Completed in Commit c57617b

  • ✅ engine/display/streaming.py - New streaming display backend with:
    • Frame compression (RLE, diff-based)
    • Binary message protocol
    • Delta frame updates for efficient streaming

This enables efficient real-time preview for the web editor.

## Completed in Commit c57617b - ✅ `engine/display/streaming.py` - New streaming display backend with: - Frame compression (RLE, diff-based) - Binary message protocol - Delta frame updates for efficient streaming This enables efficient real-time preview for the web editor.
david closed this issue 2026-03-19 20:13:19 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
epic:#2
epic:#20
epic:#3
epic:#4
epic:#5
epic:#6
epic:Data Source
epic:Display Backend
epic:Display Tools
epic:Effects
epic:Pipeline Architecture
epic:Sensors
status:feature-complete
status:in-progress
status:released
status:tested
type:epic
type:epic
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
david genewildish
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: klubhaus/sideline#26
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?