knox/docs/gossip-protocol.md

Gossip Protocol for Knox

• Status: Draft
• Date: 2026-08-29
• Scope: distributed knowledge index sharing
• Related: internal/db, internal/watch, internal/index

1. Problem & Goals

Knox currently persists a single-node SQLite index at ~/.local/share/knox/index.db. A single process tree (watch daemon + MCP clients

• web UI) shares it on one host via WAL + busy_timeout. There is no way to see knowledge collected on another machine.

Goal: let independent knox nodes (one per machine) exchange observations and converge on a shared view of what the owner learned — without requiring them to run concurrently, without a central server, and without converting the whole store into a CRDT.

Non-goals:

• Full CRDT semantics / arbitrary conflict merging of rich state.
• Realtime multi-writer collaboration on threads.
• Delete propagation / hard deletes (observations are append-only).
• Distributed search with consistency guarantees.

Consistency model: eventual consistency with a single source of truth: the append-only observations log. All materialized state (entries, threads, topics, sessions) is derived and reconcilable from it.

2. Determinism Audit

Convergence depends on determinism. Every fact knox ingests must map to the same bytes/identity on every node, or gossip cannot dedup it. Current state:

#	Nondeterminism	Where	Failure mode
A	Absolute paths in fingerprints	git.go:144 ("git:repo:"+path), log.go:72 (sha256(path)), obsidian.go:88-91 (path in meta)	Same logical fact on two hosts has different fingerprint → treated as two distinct entries → gossip can't cross-identify
B	Wall-clock as fact time	collected_at/last_seen = time.Now() (db.go:126,188); signalTime/gitStatusTime fall back to now (gitea.go:158, git.go:193)	Two nodes ingesting the same fact at different wall times produce divergent rows/sorting
C	File mtime as fact time	session.go:50-53, log.go:75-77, obsidian.go:79-81	Live files (logs) get fresh mtimes → new observations each scan even when content changed only by lines
D	TF-IDF tie-break via map iteration	tfidf.go:117-127 (TopTerms), tfidf.go:220-231 (cluster naming)	Equal scores → unspecified order → same input can yield different cluster names/membership → divergent auto-thread titles
E	Autoincrement IDs	observations.id, threads.id (schema.go:6,93)	Two nodes both auto-create a thread for the same cluster → distinct thread IDs → duplicate threads on merge
F	Order-of-arrival dedup	ObservationsByFingerprint/dedup ORDER BY id DESC (db.go:137-140,292)	Interleaved inserts from multiple nodes change which observation is "latest local" → materialized entries diverge until reconciled
G	ref_count / last_seen counters	db.go:181-189	Merge-order-dependent; must be treated as local cache, not shared truth

Verdict: content extraction is deterministic; identity, timestamps, IDs, and derived state are not. The gossip layer must fix A–D at the source and treat E–G as merge/reconcile concerns.

3. Schema Changes

3.1 observations — locator addressability

ALTER TABLE observations ADD COLUMN node_id TEXT NOT NULL DEFAULT '';
ALTER TABLE observations ADD COLUMN hcl     INTEGER;          -- Hybrid Logical Clock
ALTER TABLE observations ADD COLUMN received_at TEXT;         -- local arrival, NOT shared truth

• Primary key becomes the locator (node_id, hcl) — globally unique without coordination. The existing id AUTOINCREMENT stays only as a local rowid for unchanged code paths during migration.
• hcl is a monotonic logical clock: (wall_ms, counter) packed so that lexicographic comparison = causal order. Wall component is not trusted as a fact timestamp, only as a throttle.
• collected_at keeps meaning "when this was observed" but is stamped from the ingester's HCL, not the writer's wall clock.
• received_at is always local time.Now() — explicitly excluded from gossip.

3.2 threads — idempotent auto-creation

ALTER TABLE threads ADD COLUMN cluster_key TEXT UNIQUE;
ALTER TABLE threads ADD COLUMN hcl INTEGER;

• cluster_key = sha256 of the deterministic cluster terms (lexicographically sorted). AutoThreader computes it before deciding to create a thread (threader.go:121), and uses INSERT ... ON CONFLICT(cluster_key) DO UPDATE instead of blind INSERT. This makes auto-creation idempotent across nodes and gives the merge a stable join key.
• updated_at is migrated to an HCL; created_at/resolved_at stay descriptive.

3.3 entries — explicitly derived

No new columns. ref_count/last_seen are documented as local cache only — never gossiped, never merged. (See §6 reconcile.)

3.4 settings — peer registry

CREATE TABLE IF NOT EXISTS peers (
    peer_id       TEXT PRIMARY KEY,       -- sha256 of node keypair public part
    addr          TEXT,                   -- e.g. http://192.168.1.20:8931
    name          TEXT,
    last_handshake TEXT,
    cursor        INTEGER,               -- max hcl consumed from this peer (pull)
    created_at    TEXT DEFAULT (datetime('now'))
);

4. Determinism Fixes (prerequisite)

Land before gossip so observed facts have stable identity:

F1 — Canonical source paths. Ingesters must not fingerprint machine-local absolute paths. Change:

• SessionDiffIngester: fingerprint on content + session_id (already content-based; verified stable — no change).
• LogIngester (log.go:72): release-content-independent → fingerprint on (base filename, line count, extracted facts) or content hash of full text; SourcePath stored as filename only.
• GitIngester (git.go:144): fingerprint on "git:repo:"+remote_url+branch instead of local path.
• ObsidianIngester (obsidian.go:88-91): fingerprint on note content + relative vault path (already has relPath); drop the absolute path from meta.

F2 — HLC for all "when" storage. Replace fact-time time.Now() with the node's HLC tick in RecordObservation, entry updates, thread updates, UpsertSession. signalTime/gitStatusTime "fall back to now" become explicit unknown (""/NULL) so downstream can rebase rather than fabricate a time.

F3 — Stable TF-IDF sort. In tfidf.go, sort by (score desc, term asc) so ties are deterministic. Apply in TopTerms and cluster naming.

F4 — Locator IDs. Composite (node_id, hcl) for cross-node uniqueness. Within a single node the rowid remains monotonic, so the current dedup query (ORDER BY id DESC) can be re-expressed as ORDER BY hcl DESC.

5. Gossip Protocol

5.1 Transport

Plain HTTP/JSON on a per-node advertized address (default port 8931). Peers are seeded from a static list (KNOX_PEERS), then the swarm discovers itself: each node advertises its known peer addresses in /v1/ping, and every sweep enqueues newly-learned nodes for direct contact (membership gossip — no relay of observations). mDNS/rendezvous is future work.

Endpoints:

GET  /v1/ping                       → { node_id, name, max_hcl, peers: [{peer_id, addr, name}] }
GET  /v1/log?after=<seq>&node=<id>  → { cursor, rows: [observation...] }   # pull
POST /v1/obs/batch                  → body: [observation...]; reply: { accepted n, conflict n }  # push
GET  /v1/diff                       → divergence summary (M4)

5.2 Knowledge exchange

• Knowledge vector: each node tracks peer_id → max_hcl consumed. Anti-entropy is a pull: periodically (and on handshake) query each peer's /v1/log?after=....
• Membership gossip: /v1/ping includes the responding node's known peers (peer_id, addr, name). The caller merges them into its peers table and enqueues their addresses for direct sweeps. A new node therefore joins the whole swarm by configuring just one seed peer. Membership flows independently of data — a node never relays another's observations, only its address.
• Push: on a new local observation, best-effort POST /v1/obs/batch to known peers. A node does not re-broadcast something it merely received (that peer already has it and will pull from its origin) — this is the echo/loop suppression: ownership by node_id.
• Handshake: on discovery, GET /v1/ping, then a full pull from the peer's current cursor (i.e., since hcl=0 won't happen; use peer's max_hcl as "I have everything you have" anchor only if the peer trimmed history).

5.3 Merge rules

• Observations: append-only, idempotent via (node_id, hcl) PK. INSERT OR IGNORE. Conflicts are impossible by construction (a given node's HCL is strictly monotonic).
• Entries/threads/topics/sessions: not merged. They are rebuilt from the shared observation log by knox reconcile (§6). This keeps the log as the only replicated state and sidesteps merge-order dependence (D/F/G).
• Threads authored by humans (edited titles, motivations, notes): merged by cluster_key with LWW on thread HCL. Editing a thread bumps its HCL. thread_notes are append-only and replicated as observations-like rows if needed (deferred).

5.4 Tombstones & deletion

Observations are immutable — no deletion. For threads, closing sets resolved_at + bumps HCL; a tombstoned thread is conveyed by its LWW update, not a delete. No hard deletes except operator-initiated local cleanup.

5.5 Failure & partitions

No quorum, no leader. Writes never block on peers. A partitioned node keeps accepting local observations with its own HCL; on recovery, pull reconciles (§6). The only requirement is that each node's node_id is unique and its HCL monotonic locally — cross-node the HCL only orders causally-related rows.

5.6 Limits

• Observations are small structured rows; batch sizes keep LAN-friendly (e.g., 1000/batch). No sharding, no streaming — non-goals.
• The full log is the limit of what a peer will pull; no compaction in M3 (see Future Work).

6. Reconcile Engine

knox reconcile [--dry-run]:

1. Read all observations ordered by (node_id, hcl).
2. Rebuild entries from scratch: for each fingerprint, fold observations in HCL order (deterministic by construction) → titles/summaries/extents.
3. Recompute ref_count/last_seen as pure local derivations.
4. Re-run the intact AutoThreader idempotently via cluster_key (ON CONFLICT DO UPDATE) so threads match — including ones a peer created.
5. Reports drift (n entries would change, n threads would be added) when --dry-run.

knox gossip diff <peer> (M4) writes the set of fingerprints only the peer has, to preview what reconcile would adopt. Reconcile runs automatically after a pull; it is cheap because observations are small and the log is the single source of truth.

7. Operational Constraints

• Same-host (WAL): unchanged — multiple knox processes share one file.
• Multi-host: each host has its own index.db; gossip replicates the observation log only.
• Local caches (ref_count, last_seen, received_at) never leave the node.
• Backups: backing up the observations log of any one node is a full backup (everything else is derived).
• Node identity: node_id from a persistent generated keypair, stored in settings. Rotating it orphans old rows (acceptable; document it).

8. Milestones

M1 — Determinism fixes (prereq). DONE. F1 (canonical fingerprints), F2 (HLC in all "when" columns), F3 (TF-IDF tie-break), F4 (locator IDs + ORDER BY hcl). Verify: two fresh DBs ingesting the same real content produce identical observation hashes and identical threads (minus node_id).

M2 — Composite PK + reconcile. DONE. HCL backfilled from local rowid; (node_id, hcl) locator UNIQUE index; threads carry cluster_key (partial UNIQUE index) making auto-creation idempotent; knox reconcile rebuilds entries from the log and re-links threads. Verify: reconcile is idempotent (0 create/0 link on re-run); a log-only DB reconstructs entries and thread cluster_keys bit-identical to the original.

M3 — Peer protocol. DONE. peers table; HTTP API (GET /v1/ping with knowledge vector, GET /v1/log?node=&after= pull with cursor paging, POST /v1/obs/batch push); watch daemon serves its log and runs a periodic anti-entropy sweep (KNOX_PEERS, KNOX_PEER_ADDR), reconcile-on-pull; echo suppression by node_id ownership; knox gossip status. Verify: two nodes converge to identical logs after a bidirectional sweep (integration test internal/watch/gossip_test.go); second sweep is idempotent; time.Now()-free fact paths confirmed by grep.

M4 — Ops & UX. DONE. knox gossip subcommand: status, diff <peer-url> (observation fingerprint divergence + tombstoned auto-thread detection), and sync (one-shot sweep + reconcile). /v1/diff endpoint serves each node's knowledge summary. Gossip server starts immediately (before seed) so peers can reach a booting node. Config via env KNOX_PEER_ADDR / KNOX_PEERS. Verified e2e: two daemons, diff previewed 1655 peer-only fingerprints, sync pulled all and converged B to 1655 observations.

M5 — Prometheus metrics. DONE. /metrics served on a dedicated port (KNOX_METRICS_ADDR, default :8932, separate from gossip). Uses prometheus/client_golang; ships with the Go runtime and process collectors. Gauge set: DB-derived gauges refreshed per scrape (observations_total{source_id}, observations_last_24h, entries, projects, sessions, pending_reflections, threads_total{status}, peers, observations_by_node{node_id}, knowledge_max_hcl{node_id}) plus live gossip counters (pulls/pushes, observations_pulled/pushed, errors) incremented during the anti-entropy sweep, and knox_node_info{node_id,name} for scrape identity (tracks issue #2).

9. Future Work (explicitly out of M1–M4)

• mDNS / rendezvous peer discovery.
• Log compaction / pruning with tombstones for cutoff.
• Replicating thread_notes and multi-writer thread editing as CRDT lww-reg pairs.
• Authentication/TLS for non-trusted networks (M3 assumes trusted LAN).

10. Open Questions

• Should skills-catalog/observations from filesystem sources ever sync, or stay per-machine by design? (Default: sync all observation sources.)
• When a human edits a thread on two nodes concurrently, accept last-write-wins? (Proposal: yes for M3, revisit with CRDT lww later.)
• Do we need a heartbeat/tombstone for peer removal, or is a soft "unreachable" state enough?