knox/docs/gossip-protocol.md

# Gossip Protocol for Knox

- Status: Draft
- Date: 2026-08-29
- Scope: distributed knowledge index sharing
- Related: `internal/db`, `internal/watch`, `internal/index`

## 1. Problem & Goals

Knox currently persists a single-node SQLite index at
`~/.local/share/knox/index.db`. A single process tree (watch daemon + MCP clients
+ web UI) shares it on one host via WAL + busy_timeout. There is no way to see
knowledge collected on another machine.

**Goal:** let independent knox nodes (one per machine) exchange observations and
converge on a shared view of what the owner learned — *without* requiring them to
run concurrently, without a central server, and without converting the whole
store into a CRDT.

**Non-goals:**
- Full CRDT semantics / arbitrary conflict merging of rich state.
- Realtime multi-writer collaboration on threads.
- Delete propagation / hard deletes (observations are append-only).
- Distributed search with consistency guarantees.

**Consistency model:** eventual consistency with a single source of truth: the
append-only `observations` log. All materialized state (`entries`, threads,
topics, sessions) is *derived* and reconcilable from it.

## 2. Determinism Audit

Convergence depends on determinism. Every fact knox ingests must map to the same
bytes/identity on every node, or gossip cannot dedup it. Current state:

| # | Nondeterminism | Where | Failure mode |
|---|----------------|-------|--------------|
| A | Absolute paths in fingerprints | `git.go:144` (`"git:repo:"+path`), `log.go:72` (`sha256(path)`), `obsidian.go:88-91` (path in meta) | Same logical fact on two hosts has different fingerprint → treated as two distinct entries → gossip can't cross-identify |
| B | Wall-clock as fact time | `collected_at`/`last_seen` = `time.Now()` (`db.go:126,188`); `signalTime`/`gitStatusTime` fall back to `now` (`gitea.go:158`, `git.go:193`) | Two nodes ingesting the same fact at different wall times produce divergent rows/sorting |
| C | File mtime as fact time | `session.go:50-53`, `log.go:75-77`, `obsidian.go:79-81` | Live files (logs) get fresh mtimes → new observations each scan even when content changed only by lines |
| D | TF-IDF tie-break via map iteration | `tfidf.go:117-127` (`TopTerms`), `tfidf.go:220-231` (cluster naming) | Equal scores → unspecified order → same input can yield different cluster names/membership → divergent auto-thread titles |
| E | Autoincrement IDs | `observations.id`, `threads.id` (`schema.go:6,93`) | Two nodes both auto-create a thread for the same cluster → distinct thread IDs → duplicate threads on merge |
| F | Order-of-arrival dedup | `ObservationsByFingerprint`/dedup `ORDER BY id DESC` (`db.go:137-140,292`) | Interleaved inserts from multiple nodes change which observation is "latest local" → materialized `entries` diverge until reconciled |
| G | `ref_count` / `last_seen` counters | `db.go:181-189` | Merge-order-dependent; must be treated as local cache, not shared truth |

**Verdict:** content extraction is deterministic; identity, timestamps, IDs, and
derived state are not. The gossip layer must fix A–D at the source and treat
E–G as merge/reconcile concerns.

## 3. Schema Changes

### 3.1 `observations` — locator addressability

```sql
ALTER TABLE observations ADD COLUMN node_id TEXT NOT NULL DEFAULT '';
ALTER TABLE observations ADD COLUMN hcl     INTEGER;          -- Hybrid Logical Clock
ALTER TABLE observations ADD COLUMN received_at TEXT;         -- local arrival, NOT shared truth
```

- Primary key becomes the locator `(node_id, hcl)` — globally unique without
  coordination. The existing `id` AUTOINCREMENT stays only as a local rowid for
  unchanged code paths during migration.
- `hcl` is a monotonic logical clock: `(wall_ms, counter)` packed so that
  lexicographic comparison = causal order. Wall component is *not* trusted as a
  fact timestamp, only as a throttle.
- `collected_at` keeps meaning "when this was observed" but is stamped from the
  **ingester's** HCL, not the writer's wall clock.
- `received_at` is always local `time.Now()` — explicitly excluded from gossip.

### 3.2 `threads` — idempotent auto-creation

```sql
ALTER TABLE threads ADD COLUMN cluster_key TEXT UNIQUE;
ALTER TABLE threads ADD COLUMN hcl INTEGER;
```

- `cluster_key` = sha256 of the deterministic cluster terms (lexicographically
  sorted). AutoThreader computes it *before* deciding to create a thread
  (`threader.go:121`), and uses `INSERT ... ON CONFLICT(cluster_key) DO UPDATE`
  instead of blind `INSERT`. This makes auto-creation idempotent across nodes and
  gives the merge a stable join key.
- `updated_at` is migrated to an HCL; `created_at`/`resolved_at` stay descriptive.

### 3.3 `entries` — explicitly derived

No new columns. `ref_count`/`last_seen` are documented as **local cache only** —
never gossiped, never merged. (See §6 reconcile.)

### 3.4 `settings` — peer registry

```sql
CREATE TABLE IF NOT EXISTS peers (
    peer_id       TEXT PRIMARY KEY,       -- sha256 of node keypair public part
    addr          TEXT,                   -- e.g. http://192.168.1.20:8931
    name          TEXT,
    last_handshake TEXT,
    cursor        INTEGER,               -- max hcl consumed from this peer (pull)
    created_at    TEXT DEFAULT (datetime('now'))
);
```

## 4. Determinism Fixes (prerequisite)

Land before gossip so observed facts have stable identity:

**F1 — Canonical source paths.** Ingesters must not fingerprint machine-local
absolute paths. Change:
- `SessionDiffIngester`: fingerprint on content + session_id (already content-based; verified stable — no change).
- `LogIngester` (`log.go:72`): release-content-independent → fingerprint on (base filename, line count, extracted facts) or content hash of full text; `SourcePath` stored as filename only.
- `GitIngester` (`git.go:144`): fingerprint on `"git:repo:"+remote_url+branch` instead of local path.
- `ObsidianIngester` (`obsidian.go:88-91`): fingerprint on note content + relative vault path (already has `relPath`); drop the absolute path from meta.

**F2 — HLC for all "when" storage.** Replace fact-time `time.Now()` with the
node's HLC tick in `RecordObservation`, entry updates, thread updates,
`UpsertSession`. `signalTime`/`gitStatusTime` "fall back to now" become *explicit
unknown* (`""`/NULL) so downstream can rebase rather than fabricate a time.

**F3 — Stable TF-IDF sort.** In `tfidf.go`, sort by `(score desc, term asc)` so
ties are deterministic. Apply in `TopTerms` and cluster naming.

**F4 — Locator IDs.** Composite `(node_id, hcl)` for cross-node uniqueness. Within
a single node the rowid remains monotonic, so the current dedup query
(`ORDER BY id DESC`) can be re-expressed as `ORDER BY hcl DESC`.

## 5. Gossip Protocol

### 5.1 Transport

Plain HTTP/JSON on a per-node advertized address (default port `8931`). Nodes
discover peers via a static list in `settings` (M3). mDNS/rendezvous is future
work.

Endpoints:

```
GET  /v1/ping                       → { node_id, name, max_hcl }
GET  /v1/log?after=<seq>&node=<id>  → { cursor, rows: [observation...] }   # pull
POST /v1/obs/batch                  → body: [observation...]; reply: { accepted n, conflict n }  # push
GET  /v1/diff                       → divergence summary (M4)
```

### 5.2 Knowledge exchange

- **Knowledge vector:** each node tracks `peer_id → max_hcl consumed`. Anti-entropy
  is a pull: periodically (and on handshake) query each peer's `/v1/log?after=...`.
- **Push:** on a new local observation, best-effort `POST /v1/obs/batch` to known
  peers. A node does **not** re-broadcast something it merely received (that peer
  already has it and will pull from its origin) — this is the echo/loop
  suppression: ownership by `node_id`.
- **Handshake:** on discovery, `GET /v1/ping`, then a full pull from the peer's
  current cursor (i.e., since `hcl=0` won't happen; use peer's `max_hcl` as
  "I have everything you have" anchor only if the peer trimmed history).

### 5.3 Merge rules

- **Observations:** append-only, idempotent via `(node_id, hcl)` PK. `INSERT OR
  IGNORE`. Conflicts are impossible by construction (a given node's HCL is
  strictly monotonic).
- **Entries/threads/topics/sessions:** *not merged.* They are rebuilt from the
  shared observation log by `knox reconcile` (§6). This keeps the log as the only
  replicated state and sidesteps merge-order dependence (D/F/G).
- **Threads authored by humans** (edited titles, motivations, notes): merged by
  `cluster_key` with LWW on thread HCL. Editing a thread bumps its HCL. `thread_notes`
  are append-only and replicated as observations-like rows if needed (deferred).

### 5.4 Tombstones & deletion

Observations are immutable — no deletion. For threads, closing sets
`resolved_at` + bumps HCL; a tombstoned thread is conveyed by its LWW update, not
a delete. No hard deletes except operator-initiated local cleanup.

### 5.5 Failure & partitions

No quorum, no leader. Writes never block on peers. A partitioned node keeps
accepting local observations with its own HCL; on recovery, pull reconciles
(§6). The only requirement is that each node's `node_id` is unique and its HCL
monotonic *locally* — cross-node the HCL only orders causally-related rows.

### 5.6 Limits

- Observations are small structured rows; batch sizes keep LAN-friendly (e.g.,
  1000/batch). No sharding, no streaming — non-goals.
- The full log is the limit of what a peer will pull; no compaction in M3 (see
  Future Work).

## 6. Reconcile Engine

`knox reconcile [--dry-run]`:

1. Read all observations ordered by `(node_id, hcl)`.
2. Rebuild `entries` from scratch: for each fingerprint, fold observations in
   HCL order (deterministic by construction) → titles/summaries/extents.
3. Recompute `ref_count`/`last_seen` as pure local derivations.
4. Re-run the intact AutoThreader idempotently via `cluster_key`
   (ON CONFLICT DO UPDATE) so threads match — including ones a peer created.
5. Reports drift (`n entries would change`, `n threads would be added`) when
   `--dry-run`.

`knox gossip diff <peer>` (M4) writes the set of fingerprints only the peer has,
to preview what reconcile would adopt. Reconcile runs automatically after a pull;
it is cheap because observations are small and the log is the single source of
truth.

## 7. Operational Constraints

- **Same-host (WAL):** unchanged — multiple knox processes share one file.
- **Multi-host:** each host has its own `index.db`; gossip replicates the
  *observation log only*.
- **Local caches** (`ref_count`, `last_seen`, `received_at`) never leave the node.
- **Backups:** backing up the observations log of any one node is a full backup
  (everything else is derived).
- **Node identity:** `node_id` from a persistent generated keypair, stored in
  `settings`. Rotating it orphans old rows (acceptable; document it).

## 8. Milestones

**M1 — Determinism fixes (prereq).** DONE. F1 (canonical fingerprints), F2 (HLC
in all "when" columns), F3 (TF-IDF tie-break), F4 (locator IDs + `ORDER BY hcl`).
Verify: two fresh DBs ingesting the same real content produce identical
observation hashes and identical threads (minus node_id).

**M2 — Composite PK + reconcile.** DONE. HCL backfilled from local rowid;
`(node_id, hcl)` locator UNIQUE index; threads carry `cluster_key` (partial
UNIQUE index) making auto-creation idempotent; `knox reconcile` rebuilds
`entries` from the log and re-links threads. Verify: reconcile is idempotent (0
create/0 link on re-run); a log-only DB reconstructs `entries` and thread
`cluster_key`s bit-identical to the original.

**M3 — Peer protocol.** DONE. `peers` table; HTTP API (`GET /v1/ping` with
knowledge vector, `GET /v1/log?node=&after=` pull with cursor paging,
`POST /v1/obs/batch` push); `watch` daemon serves its log and runs a periodic
anti-entropy sweep (`KNOX_PEERS`, `KNOX_PEER_ADDR`), reconcile-on-pull; echo
suppression by `node_id` ownership; `knox gossip status`. Verify: two nodes
converge to identical logs after a bidirectional sweep (integration test
`internal/watch/gossip_test.go`); second sweep is idempotent; `time.Now()`-free
fact paths confirmed by grep.

**M4 — Ops & UX.** `knox gossip` subcommand (status/diff), reconcile-on-pull,
tombstoned thread handling in diff output, logging, config (env `KNOX_PEER_ADDR`,
`KNOX_PEERS`).

## 9. Future Work (explicitly out of M1–M4)

- mDNS / rendezvous peer discovery.
- Log compaction / pruning with tombstones for cutoff.
- Replicating `thread_notes` and multi-writer thread editing as CRDT lww-reg
  pairs.
- Authentication/TLS for non-trusted networks (M3 assumes trusted LAN).

## 10. Open Questions

- Should `skills-catalog`/`observations` from `filesystem` sources ever sync, or
  stay per-machine by design? (Default: sync all observation sources.)
- When a human edits a thread on two nodes concurrently, accept last-write-wins?
  (Proposal: yes for M3, revisit with CRDT lww later.)
- Do we need a heartbeat/tombstone for *peer* removal, or is a soft "unreachable"
  state enough?