diff --git a/docs/gossip-protocol.md b/docs/gossip-protocol.md new file mode 100644 index 0000000..83f6b88 --- /dev/null +++ b/docs/gossip-protocol.md @@ -0,0 +1,249 @@ +# Gossip Protocol for Knox + +- Status: Draft +- Date: 2026-08-29 +- Scope: distributed knowledge index sharing +- Related: `internal/db`, `internal/watch`, `internal/index` + +## 1. Problem & Goals + +Knox currently persists a single-node SQLite index at +`~/.local/share/knox/index.db`. A single process tree (watch daemon + MCP clients ++ web UI) shares it on one host via WAL + busy_timeout. There is no way to see +knowledge collected on another machine. + +**Goal:** let independent knox nodes (one per machine) exchange observations and +converge on a shared view of what the owner learned — *without* requiring them to +run concurrently, without a central server, and without converting the whole +store into a CRDT. + +**Non-goals:** +- Full CRDT semantics / arbitrary conflict merging of rich state. +- Realtime multi-writer collaboration on threads. +- Delete propagation / hard deletes (observations are append-only). +- Distributed search with consistency guarantees. + +**Consistency model:** eventual consistency with a single source of truth: the +append-only `observations` log. All materialized state (`entries`, threads, +topics, sessions) is *derived* and reconcilable from it. + +## 2. Determinism Audit + +Convergence depends on determinism. Every fact knox ingests must map to the same +bytes/identity on every node, or gossip cannot dedup it. Current state: + +| # | Nondeterminism | Where | Failure mode | +|---|----------------|-------|--------------| +| A | Absolute paths in fingerprints | `git.go:144` (`"git:repo:"+path`), `log.go:72` (`sha256(path)`), `obsidian.go:88-91` (path in meta) | Same logical fact on two hosts has different fingerprint → treated as two distinct entries → gossip can't cross-identify | +| B | Wall-clock as fact time | `collected_at`/`last_seen` = `time.Now()` (`db.go:126,188`); `signalTime`/`gitStatusTime` fall back to `now` (`gitea.go:158`, `git.go:193`) | Two nodes ingesting the same fact at different wall times produce divergent rows/sorting | +| C | File mtime as fact time | `session.go:50-53`, `log.go:75-77`, `obsidian.go:79-81` | Live files (logs) get fresh mtimes → new observations each scan even when content changed only by lines | +| D | TF-IDF tie-break via map iteration | `tfidf.go:117-127` (`TopTerms`), `tfidf.go:220-231` (cluster naming) | Equal scores → unspecified order → same input can yield different cluster names/membership → divergent auto-thread titles | +| E | Autoincrement IDs | `observations.id`, `threads.id` (`schema.go:6,93`) | Two nodes both auto-create a thread for the same cluster → distinct thread IDs → duplicate threads on merge | +| F | Order-of-arrival dedup | `ObservationsByFingerprint`/dedup `ORDER BY id DESC` (`db.go:137-140,292`) | Interleaved inserts from multiple nodes change which observation is "latest local" → materialized `entries` diverge until reconciled | +| G | `ref_count` / `last_seen` counters | `db.go:181-189` | Merge-order-dependent; must be treated as local cache, not shared truth | + +**Verdict:** content extraction is deterministic; identity, timestamps, IDs, and +derived state are not. The gossip layer must fix A–D at the source and treat +E–G as merge/reconcile concerns. + +## 3. Schema Changes + +### 3.1 `observations` — locator addressability + +```sql +ALTER TABLE observations ADD COLUMN node_id TEXT NOT NULL DEFAULT ''; +ALTER TABLE observations ADD COLUMN hcl INTEGER; -- Hybrid Logical Clock +ALTER TABLE observations ADD COLUMN received_at TEXT; -- local arrival, NOT shared truth +``` + +- Primary key becomes the locator `(node_id, hcl)` — globally unique without + coordination. The existing `id` AUTOINCREMENT stays only as a local rowid for + unchanged code paths during migration. +- `hcl` is a monotonic logical clock: `(wall_ms, counter)` packed so that + lexicographic comparison = causal order. Wall component is *not* trusted as a + fact timestamp, only as a throttle. +- `collected_at` keeps meaning "when this was observed" but is stamped from the + **ingester's** HCL, not the writer's wall clock. +- `received_at` is always local `time.Now()` — explicitly excluded from gossip. + +### 3.2 `threads` — idempotent auto-creation + +```sql +ALTER TABLE threads ADD COLUMN cluster_key TEXT UNIQUE; +ALTER TABLE threads ADD COLUMN hcl INTEGER; +``` + +- `cluster_key` = sha256 of the deterministic cluster terms (lexicographically + sorted). AutoThreader computes it *before* deciding to create a thread + (`threader.go:121`), and uses `INSERT ... ON CONFLICT(cluster_key) DO UPDATE` + instead of blind `INSERT`. This makes auto-creation idempotent across nodes and + gives the merge a stable join key. +- `updated_at` is migrated to an HCL; `created_at`/`resolved_at` stay descriptive. + +### 3.3 `entries` — explicitly derived + +No new columns. `ref_count`/`last_seen` are documented as **local cache only** — +never gossiped, never merged. (See §6 reconcile.) + +### 3.4 `settings` — peer registry + +```sql +CREATE TABLE IF NOT EXISTS peers ( + peer_id TEXT PRIMARY KEY, -- sha256 of node keypair public part + addr TEXT, -- e.g. http://192.168.1.20:8931 + name TEXT, + last_handshake TEXT, + cursor INTEGER, -- max hcl consumed from this peer (pull) + created_at TEXT DEFAULT (datetime('now')) +); +``` + +## 4. Determinism Fixes (prerequisite) + +Land before gossip so observed facts have stable identity: + +**F1 — Canonical source paths.** Ingesters must not fingerprint machine-local +absolute paths. Change: +- `SessionDiffIngester`: fingerprint on content + session_id (already content-based; verified stable — no change). +- `LogIngester` (`log.go:72`): release-content-independent → fingerprint on (base filename, line count, extracted facts) or content hash of full text; `SourcePath` stored as filename only. +- `GitIngester` (`git.go:144`): fingerprint on `"git:repo:"+remote_url+branch` instead of local path. +- `ObsidianIngester` (`obsidian.go:88-91`): fingerprint on note content + relative vault path (already has `relPath`); drop the absolute path from meta. + +**F2 — HLC for all "when" storage.** Replace fact-time `time.Now()` with the +node's HLC tick in `RecordObservation`, entry updates, thread updates, +`UpsertSession`. `signalTime`/`gitStatusTime` "fall back to now" become *explicit +unknown* (`""`/NULL) so downstream can rebase rather than fabricate a time. + +**F3 — Stable TF-IDF sort.** In `tfidf.go`, sort by `(score desc, term asc)` so +ties are deterministic. Apply in `TopTerms` and cluster naming. + +**F4 — Locator IDs.** Composite `(node_id, hcl)` for cross-node uniqueness. Within +a single node the rowid remains monotonic, so the current dedup query +(`ORDER BY id DESC`) can be re-expressed as `ORDER BY hcl DESC`. + +## 5. Gossip Protocol + +### 5.1 Transport + +Plain HTTP/JSON on a per-node advertized address (default port `8931`). Nodes +discover peers via a static list in `settings` (M3). mDNS/rendezvous is future +work. + +Endpoints: + +``` +GET /v1/ping → { node_id, name, max_hcl } +GET /v1/log?after=&node= → { cursor, rows: [observation...] } # pull +POST /v1/obs/batch → body: [observation...]; reply: { accepted n, conflict n } # push +GET /v1/diff → divergence summary (M4) +``` + +### 5.2 Knowledge exchange + +- **Knowledge vector:** each node tracks `peer_id → max_hcl consumed`. Anti-entropy + is a pull: periodically (and on handshake) query each peer's `/v1/log?after=...`. +- **Push:** on a new local observation, best-effort `POST /v1/obs/batch` to known + peers. A node does **not** re-broadcast something it merely received (that peer + already has it and will pull from its origin) — this is the echo/loop + suppression: ownership by `node_id`. +- **Handshake:** on discovery, `GET /v1/ping`, then a full pull from the peer's + current cursor (i.e., since `hcl=0` won't happen; use peer's `max_hcl` as + "I have everything you have" anchor only if the peer trimmed history). + +### 5.3 Merge rules + +- **Observations:** append-only, idempotent via `(node_id, hcl)` PK. `INSERT OR + IGNORE`. Conflicts are impossible by construction (a given node's HCL is + strictly monotonic). +- **Entries/threads/topics/sessions:** *not merged.* They are rebuilt from the + shared observation log by `knox reconcile` (§6). This keeps the log as the only + replicated state and sidesteps merge-order dependence (D/F/G). +- **Threads authored by humans** (edited titles, motivations, notes): merged by + `cluster_key` with LWW on thread HCL. Editing a thread bumps its HCL. `thread_notes` + are append-only and replicated as observations-like rows if needed (deferred). + +### 5.4 Tombstones & deletion + +Observations are immutable — no deletion. For threads, closing sets +`resolved_at` + bumps HCL; a tombstoned thread is conveyed by its LWW update, not +a delete. No hard deletes except operator-initiated local cleanup. + +### 5.5 Failure & partitions + +No quorum, no leader. Writes never block on peers. A partitioned node keeps +accepting local observations with its own HCL; on recovery, pull reconciles +(§6). The only requirement is that each node's `node_id` is unique and its HCL +monotonic *locally* — cross-node the HCL only orders causally-related rows. + +### 5.6 Limits + +- Observations are small structured rows; batch sizes keep LAN-friendly (e.g., + 1000/batch). No sharding, no streaming — non-goals. +- The full log is the limit of what a peer will pull; no compaction in M3 (see + Future Work). + +## 6. Reconcile Engine + +`knox reconcile [--dry-run]`: + +1. Read all observations ordered by `(node_id, hcl)`. +2. Rebuild `entries` from scratch: for each fingerprint, fold observations in + HCL order (deterministic by construction) → titles/summaries/extents. +3. Recompute `ref_count`/`last_seen` as pure local derivations. +4. Re-run the intact AutoThreader idempotently via `cluster_key` + (ON CONFLICT DO UPDATE) so threads match — including ones a peer created. +5. Reports drift (`n entries would change`, `n threads would be added`) when + `--dry-run`. + +`knox gossip diff ` (M4) writes the set of fingerprints only the peer has, +to preview what reconcile would adopt. Reconcile runs automatically after a pull; +it is cheap because observations are small and the log is the single source of +truth. + +## 7. Operational Constraints + +- **Same-host (WAL):** unchanged — multiple knox processes share one file. +- **Multi-host:** each host has its own `index.db`; gossip replicates the + *observation log only*. +- **Local caches** (`ref_count`, `last_seen`, `received_at`) never leave the node. +- **Backups:** backing up the observations log of any one node is a full backup + (everything else is derived). +- **Node identity:** `node_id` from a persistent generated keypair, stored in + `settings`. Rotating it orphans old rows (acceptable; document it). + +## 8. Milestones + +**M1 — Determinism fixes (prereq).** F1 (canonical fingerprints), F2 (HLC in all +"when" columns), F3 (TF-IDF tie-break), F4 (locator IDs + `ORDER BY hcl`). +Verify: two fresh DBs ingesting the same real content produce identical +observation hashes and identical threads (minus node_id). + +**M2 — Composite PK + reconcile.** Schema migration; `knox reconcile`; `entries` +fully derived; dedup re-expressed on HCL. Verify: reconcile is idempotent; a DB +with only the log reconstructs `entries`/threads bit-identical to the original. + +**M3 — Peer protocol.** `peers` settings table, `/v1/ping`, `/v1/log` pull, +`/v1/obs/batch` push, handshake + periodic anti-entropy, echo suppression. +Verify: two nodes converge to identical logs after partition (integration test). +`time.Now()`-free fact paths confirmed by grep. + +**M4 — Ops & UX.** `knox gossip` subcommand (status/diff), reconcile-on-pull, +tombstoned thread handling in diff output, logging, config (env `KNOX_PEER_ADDR`, +`KNOX_PEERS`). + +## 9. Future Work (explicitly out of M1–M4) + +- mDNS / rendezvous peer discovery. +- Log compaction / pruning with tombstones for cutoff. +- Replicating `thread_notes` and multi-writer thread editing as CRDT lww-reg + pairs. +- Authentication/TLS for non-trusted networks (M3 assumes trusted LAN). + +## 10. Open Questions + +- Should `skills-catalog`/`observations` from `filesystem` sources ever sync, or + stay per-machine by design? (Default: sync all observation sources.) +- When a human edits a thread on two nodes concurrently, accept last-write-wins? + (Proposal: yes for M3, revisit with CRDT lww later.) +- Do we need a heartbeat/tombstone for *peer* removal, or is a soft "unreachable" + state enough? \ No newline at end of file