docs: gossip protocol design for cross-machine sharing

Tracks issue #1. Audits knox determinism (path-based fingerprints, wall-clock fact times, TF-IDF tie-breaks, autoincrement IDs) and specifies schema changes, HLC clocks, merge rules, reconcile, and M1-M4 milestones.
2026-08-29 03:40:53 -07:00
parent 8eecde18b4
commit f5766aa6d8
1 changed files with 249 additions and 0 deletions
@@ -0,0 +1,249 @@
 # Gossip Protocol for Knox
 - Status: Draft
 - Date: 2026-08-29
 - Scope: distributed knowledge index sharing
 - Related: `internal/db`, `internal/watch`, `internal/index`
 ## 1. Problem & Goals
 Knox currently persists a single-node SQLite index at
 `~/.local/share/knox/index.db`. A single process tree (watch daemon + MCP clients
 + web UI) shares it on one host via WAL + busy_timeout. There is no way to see
 knowledge collected on another machine.
 **Goal:** let independent knox nodes (one per machine) exchange observations and
 converge on a shared view of what the owner learned — *without* requiring them to
 run concurrently, without a central server, and without converting the whole
 store into a CRDT.
 **Non-goals:**
 - Full CRDT semantics / arbitrary conflict merging of rich state.
 - Realtime multi-writer collaboration on threads.
 - Delete propagation / hard deletes (observations are append-only).
 - Distributed search with consistency guarantees.
 **Consistency model:** eventual consistency with a single source of truth: the
 append-only `observations` log. All materialized state (`entries`, threads,
 topics, sessions) is *derived* and reconcilable from it.
 ## 2. Determinism Audit
 Convergence depends on determinism. Every fact knox ingests must map to the same
 bytes/identity on every node, or gossip cannot dedup it. Current state:
 | # | Nondeterminism | Where | Failure mode |
 |---|----------------|-------|--------------|
 | A | Absolute paths in fingerprints | `git.go:144` (`"git:repo:"+path`), `log.go:72` (`sha256(path)`), `obsidian.go:88-91` (path in meta) | Same logical fact on two hosts has different fingerprint → treated as two distinct entries → gossip can't cross-identify |
 | B | Wall-clock as fact time | `collected_at`/`last_seen` = `time.Now()` (`db.go:126,188`); `signalTime`/`gitStatusTime` fall back to `now` (`gitea.go:158`, `git.go:193`) | Two nodes ingesting the same fact at different wall times produce divergent rows/sorting |
 | C | File mtime as fact time | `session.go:50-53`, `log.go:75-77`, `obsidian.go:79-81` | Live files (logs) get fresh mtimes → new observations each scan even when content changed only by lines |
 | D | TF-IDF tie-break via map iteration | `tfidf.go:117-127` (`TopTerms`), `tfidf.go:220-231` (cluster naming) | Equal scores → unspecified order → same input can yield different cluster names/membership → divergent auto-thread titles |
 | E | Autoincrement IDs | `observations.id`, `threads.id` (`schema.go:6,93`) | Two nodes both auto-create a thread for the same cluster → distinct thread IDs → duplicate threads on merge |
 | F | Order-of-arrival dedup | `ObservationsByFingerprint`/dedup `ORDER BY id DESC` (`db.go:137-140,292`) | Interleaved inserts from multiple nodes change which observation is "latest local" → materialized `entries` diverge until reconciled |
 | G | `ref_count` / `last_seen` counters | `db.go:181-189` | Merge-order-dependent; must be treated as local cache, not shared truth |
 **Verdict:** content extraction is deterministic; identity, timestamps, IDs, and
 derived state are not. The gossip layer must fix A–D at the source and treat
 E–G as merge/reconcile concerns.
 ## 3. Schema Changes
 ### 3.1 `observations` — locator addressability
 ```sql
 ALTER TABLE observations ADD COLUMN node_id TEXT NOT NULL DEFAULT '';
 ALTER TABLE observations ADD COLUMN hcl     INTEGER;          -- Hybrid Logical Clock
 ALTER TABLE observations ADD COLUMN received_at TEXT;         -- local arrival, NOT shared truth
 ```
 - Primary key becomes the locator `(node_id, hcl)` — globally unique without
  coordination. The existing `id` AUTOINCREMENT stays only as a local rowid for
  unchanged code paths during migration.
 - `hcl` is a monotonic logical clock: `(wall_ms, counter)` packed so that
  lexicographic comparison = causal order. Wall component is *not* trusted as a
  fact timestamp, only as a throttle.
 - `collected_at` keeps meaning "when this was observed" but is stamped from the
  **ingester's** HCL, not the writer's wall clock.
 - `received_at` is always local `time.Now()` — explicitly excluded from gossip.
 ### 3.2 `threads` — idempotent auto-creation
 ```sql
 ALTER TABLE threads ADD COLUMN cluster_key TEXT UNIQUE;
 ALTER TABLE threads ADD COLUMN hcl INTEGER;
 ```
 - `cluster_key` = sha256 of the deterministic cluster terms (lexicographically
  sorted). AutoThreader computes it *before* deciding to create a thread
  (`threader.go:121`), and uses `INSERT ... ON CONFLICT(cluster_key) DO UPDATE`
  instead of blind `INSERT`. This makes auto-creation idempotent across nodes and
  gives the merge a stable join key.
 - `updated_at` is migrated to an HCL; `created_at`/`resolved_at` stay descriptive.
 ### 3.3 `entries` — explicitly derived
 No new columns. `ref_count`/`last_seen` are documented as **local cache only** —
 never gossiped, never merged. (See §6 reconcile.)
 ### 3.4 `settings` — peer registry
 ```sql
 CREATE TABLE IF NOT EXISTS peers (
    peer_id       TEXT PRIMARY KEY,       -- sha256 of node keypair public part
    addr          TEXT,                   -- e.g. http://192.168.1.20:8931
    name          TEXT,
    last_handshake TEXT,
    cursor        INTEGER,               -- max hcl consumed from this peer (pull)
    created_at    TEXT DEFAULT (datetime('now'))
 );
 ```
 ## 4. Determinism Fixes (prerequisite)
 Land before gossip so observed facts have stable identity:
 **F1 — Canonical source paths.** Ingesters must not fingerprint machine-local
 absolute paths. Change:
 - `SessionDiffIngester`: fingerprint on content + session_id (already content-based; verified stable — no change).
 - `LogIngester` (`log.go:72`): release-content-independent → fingerprint on (base filename, line count, extracted facts) or content hash of full text; `SourcePath` stored as filename only.
 - `GitIngester` (`git.go:144`): fingerprint on `"git:repo:"+remote_url+branch` instead of local path.
 - `ObsidianIngester` (`obsidian.go:88-91`): fingerprint on note content + relative vault path (already has `relPath`); drop the absolute path from meta.
 **F2 — HLC for all "when" storage.** Replace fact-time `time.Now()` with the
 node's HLC tick in `RecordObservation`, entry updates, thread updates,
 `UpsertSession`. `signalTime`/`gitStatusTime` "fall back to now" become *explicit
 unknown* (`""`/NULL) so downstream can rebase rather than fabricate a time.
 **F3 — Stable TF-IDF sort.** In `tfidf.go`, sort by `(score desc, term asc)` so
 ties are deterministic. Apply in `TopTerms` and cluster naming.
 **F4 — Locator IDs.** Composite `(node_id, hcl)` for cross-node uniqueness. Within
 a single node the rowid remains monotonic, so the current dedup query
 (`ORDER BY id DESC`) can be re-expressed as `ORDER BY hcl DESC`.
 ## 5. Gossip Protocol
 ### 5.1 Transport
 Plain HTTP/JSON on a per-node advertized address (default port `8931`). Nodes
 discover peers via a static list in `settings` (M3). mDNS/rendezvous is future
 work.
 Endpoints:
 ```
 GET  /v1/ping                       → { node_id, name, max_hcl }
 GET  /v1/log?after=<seq>&node=<id>  → { cursor, rows: [observation...] }   # pull
 POST /v1/obs/batch                  → body: [observation...]; reply: { accepted n, conflict n }  # push
 GET  /v1/diff                       → divergence summary (M4)
 ```
 ### 5.2 Knowledge exchange
 - **Knowledge vector:** each node tracks `peer_id → max_hcl consumed`. Anti-entropy
  is a pull: periodically (and on handshake) query each peer's `/v1/log?after=...`.
 - **Push:** on a new local observation, best-effort `POST /v1/obs/batch` to known
  peers. A node does **not** re-broadcast something it merely received (that peer
  already has it and will pull from its origin) — this is the echo/loop
  suppression: ownership by `node_id`.
 - **Handshake:** on discovery, `GET /v1/ping`, then a full pull from the peer's
  current cursor (i.e., since `hcl=0` won't happen; use peer's `max_hcl` as
  "I have everything you have" anchor only if the peer trimmed history).
 ### 5.3 Merge rules
 - **Observations:** append-only, idempotent via `(node_id, hcl)` PK. `INSERT OR
  IGNORE`. Conflicts are impossible by construction (a given node's HCL is
  strictly monotonic).
 - **Entries/threads/topics/sessions:** *not merged.* They are rebuilt from the
  shared observation log by `knox reconcile` (§6). This keeps the log as the only
  replicated state and sidesteps merge-order dependence (D/F/G).
 - **Threads authored by humans** (edited titles, motivations, notes): merged by
  `cluster_key` with LWW on thread HCL. Editing a thread bumps its HCL. `thread_notes`
  are append-only and replicated as observations-like rows if needed (deferred).
 ### 5.4 Tombstones & deletion
 Observations are immutable — no deletion. For threads, closing sets
 `resolved_at` + bumps HCL; a tombstoned thread is conveyed by its LWW update, not
 a delete. No hard deletes except operator-initiated local cleanup.
 ### 5.5 Failure & partitions
 No quorum, no leader. Writes never block on peers. A partitioned node keeps
 accepting local observations with its own HCL; on recovery, pull reconciles
 (§6). The only requirement is that each node's `node_id` is unique and its HCL
 monotonic *locally* — cross-node the HCL only orders causally-related rows.
 ### 5.6 Limits
 - Observations are small structured rows; batch sizes keep LAN-friendly (e.g.,
  1000/batch). No sharding, no streaming — non-goals.
 - The full log is the limit of what a peer will pull; no compaction in M3 (see
  Future Work).
 ## 6. Reconcile Engine
 `knox reconcile [--dry-run]`:
 1. Read all observations ordered by `(node_id, hcl)`.
 2. Rebuild `entries` from scratch: for each fingerprint, fold observations in
   HCL order (deterministic by construction) → titles/summaries/extents.
 3. Recompute `ref_count`/`last_seen` as pure local derivations.
 4. Re-run the intact AutoThreader idempotently via `cluster_key`
   (ON CONFLICT DO UPDATE) so threads match — including ones a peer created.
 5. Reports drift (`n entries would change`, `n threads would be added`) when
   `--dry-run`.
 `knox gossip diff <peer>` (M4) writes the set of fingerprints only the peer has,
 to preview what reconcile would adopt. Reconcile runs automatically after a pull;
 it is cheap because observations are small and the log is the single source of
 truth.
 ## 7. Operational Constraints
 - **Same-host (WAL):** unchanged — multiple knox processes share one file.
 - **Multi-host:** each host has its own `index.db`; gossip replicates the
  *observation log only*.
 - **Local caches** (`ref_count`, `last_seen`, `received_at`) never leave the node.
 - **Backups:** backing up the observations log of any one node is a full backup
  (everything else is derived).
 - **Node identity:** `node_id` from a persistent generated keypair, stored in
  `settings`. Rotating it orphans old rows (acceptable; document it).
 ## 8. Milestones
 **M1 — Determinism fixes (prereq).** F1 (canonical fingerprints), F2 (HLC in all
 "when" columns), F3 (TF-IDF tie-break), F4 (locator IDs + `ORDER BY hcl`).
 Verify: two fresh DBs ingesting the same real content produce identical
 observation hashes and identical threads (minus node_id).
 **M2 — Composite PK + reconcile.** Schema migration; `knox reconcile`; `entries`
 fully derived; dedup re-expressed on HCL. Verify: reconcile is idempotent; a DB
 with only the log reconstructs `entries`/threads bit-identical to the original.
 **M3 — Peer protocol.** `peers` settings table, `/v1/ping`, `/v1/log` pull,
 `/v1/obs/batch` push, handshake + periodic anti-entropy, echo suppression.
 Verify: two nodes converge to identical logs after partition (integration test).
 `time.Now()`-free fact paths confirmed by grep.
 **M4 — Ops & UX.** `knox gossip` subcommand (status/diff), reconcile-on-pull,
 tombstoned thread handling in diff output, logging, config (env `KNOX_PEER_ADDR`,
 `KNOX_PEERS`).
 ## 9. Future Work (explicitly out of M1–M4)
 - mDNS / rendezvous peer discovery.
 - Log compaction / pruning with tombstones for cutoff.
 - Replicating `thread_notes` and multi-writer thread editing as CRDT lww-reg
  pairs.
 - Authentication/TLS for non-trusted networks (M3 assumes trusted LAN).
 ## 10. Open Questions
 - Should `skills-catalog`/`observations` from `filesystem` sources ever sync, or
  stay per-machine by design? (Default: sync all observation sources.)
 - When a human edits a thread on two nodes concurrently, accept last-write-wins?
  (Proposal: yes for M3, revisit with CRDT lww later.)
 - Do we need a heartbeat/tombstone for *peer* removal, or is a soft "unreachable"
  state enough?