david
25a7112d8a
Refs #1 - /v1/ping now advertises the node's known peers (peer_id, addr, name) - Run sweeps static KNOX_PEERS + persisted discovered peers, enqueueing newly-learned addresses for direct sweeps (membership-only relay; no observation relay) - db: ShareablePeers, SwarmPeerAddrs, MergePeer (cursor-preserving discovery upsert), MaxHCLForNode - integration test: a node configured with a single seed discovers and pulls from other swarm members without direct configuration
267 lines
14 KiB
Markdown
267 lines
14 KiB
Markdown
# Gossip Protocol for Knox
|
||
|
||
- Status: Draft
|
||
- Date: 2026-08-29
|
||
- Scope: distributed knowledge index sharing
|
||
- Related: `internal/db`, `internal/watch`, `internal/index`
|
||
|
||
## 1. Problem & Goals
|
||
|
||
Knox currently persists a single-node SQLite index at
|
||
`~/.local/share/knox/index.db`. A single process tree (watch daemon + MCP clients
|
||
+ web UI) shares it on one host via WAL + busy_timeout. There is no way to see
|
||
knowledge collected on another machine.
|
||
|
||
**Goal:** let independent knox nodes (one per machine) exchange observations and
|
||
converge on a shared view of what the owner learned — *without* requiring them to
|
||
run concurrently, without a central server, and without converting the whole
|
||
store into a CRDT.
|
||
|
||
**Non-goals:**
|
||
- Full CRDT semantics / arbitrary conflict merging of rich state.
|
||
- Realtime multi-writer collaboration on threads.
|
||
- Delete propagation / hard deletes (observations are append-only).
|
||
- Distributed search with consistency guarantees.
|
||
|
||
**Consistency model:** eventual consistency with a single source of truth: the
|
||
append-only `observations` log. All materialized state (`entries`, threads,
|
||
topics, sessions) is *derived* and reconcilable from it.
|
||
|
||
## 2. Determinism Audit
|
||
|
||
Convergence depends on determinism. Every fact knox ingests must map to the same
|
||
bytes/identity on every node, or gossip cannot dedup it. Current state:
|
||
|
||
| # | Nondeterminism | Where | Failure mode |
|
||
|---|----------------|-------|--------------|
|
||
| A | Absolute paths in fingerprints | `git.go:144` (`"git:repo:"+path`), `log.go:72` (`sha256(path)`), `obsidian.go:88-91` (path in meta) | Same logical fact on two hosts has different fingerprint → treated as two distinct entries → gossip can't cross-identify |
|
||
| B | Wall-clock as fact time | `collected_at`/`last_seen` = `time.Now()` (`db.go:126,188`); `signalTime`/`gitStatusTime` fall back to `now` (`gitea.go:158`, `git.go:193`) | Two nodes ingesting the same fact at different wall times produce divergent rows/sorting |
|
||
| C | File mtime as fact time | `session.go:50-53`, `log.go:75-77`, `obsidian.go:79-81` | Live files (logs) get fresh mtimes → new observations each scan even when content changed only by lines |
|
||
| D | TF-IDF tie-break via map iteration | `tfidf.go:117-127` (`TopTerms`), `tfidf.go:220-231` (cluster naming) | Equal scores → unspecified order → same input can yield different cluster names/membership → divergent auto-thread titles |
|
||
| E | Autoincrement IDs | `observations.id`, `threads.id` (`schema.go:6,93`) | Two nodes both auto-create a thread for the same cluster → distinct thread IDs → duplicate threads on merge |
|
||
| F | Order-of-arrival dedup | `ObservationsByFingerprint`/dedup `ORDER BY id DESC` (`db.go:137-140,292`) | Interleaved inserts from multiple nodes change which observation is "latest local" → materialized `entries` diverge until reconciled |
|
||
| G | `ref_count` / `last_seen` counters | `db.go:181-189` | Merge-order-dependent; must be treated as local cache, not shared truth |
|
||
|
||
**Verdict:** content extraction is deterministic; identity, timestamps, IDs, and
|
||
derived state are not. The gossip layer must fix A–D at the source and treat
|
||
E–G as merge/reconcile concerns.
|
||
|
||
## 3. Schema Changes
|
||
|
||
### 3.1 `observations` — locator addressability
|
||
|
||
```sql
|
||
ALTER TABLE observations ADD COLUMN node_id TEXT NOT NULL DEFAULT '';
|
||
ALTER TABLE observations ADD COLUMN hcl INTEGER; -- Hybrid Logical Clock
|
||
ALTER TABLE observations ADD COLUMN received_at TEXT; -- local arrival, NOT shared truth
|
||
```
|
||
|
||
- Primary key becomes the locator `(node_id, hcl)` — globally unique without
|
||
coordination. The existing `id` AUTOINCREMENT stays only as a local rowid for
|
||
unchanged code paths during migration.
|
||
- `hcl` is a monotonic logical clock: `(wall_ms, counter)` packed so that
|
||
lexicographic comparison = causal order. Wall component is *not* trusted as a
|
||
fact timestamp, only as a throttle.
|
||
- `collected_at` keeps meaning "when this was observed" but is stamped from the
|
||
**ingester's** HCL, not the writer's wall clock.
|
||
- `received_at` is always local `time.Now()` — explicitly excluded from gossip.
|
||
|
||
### 3.2 `threads` — idempotent auto-creation
|
||
|
||
```sql
|
||
ALTER TABLE threads ADD COLUMN cluster_key TEXT UNIQUE;
|
||
ALTER TABLE threads ADD COLUMN hcl INTEGER;
|
||
```
|
||
|
||
- `cluster_key` = sha256 of the deterministic cluster terms (lexicographically
|
||
sorted). AutoThreader computes it *before* deciding to create a thread
|
||
(`threader.go:121`), and uses `INSERT ... ON CONFLICT(cluster_key) DO UPDATE`
|
||
instead of blind `INSERT`. This makes auto-creation idempotent across nodes and
|
||
gives the merge a stable join key.
|
||
- `updated_at` is migrated to an HCL; `created_at`/`resolved_at` stay descriptive.
|
||
|
||
### 3.3 `entries` — explicitly derived
|
||
|
||
No new columns. `ref_count`/`last_seen` are documented as **local cache only** —
|
||
never gossiped, never merged. (See §6 reconcile.)
|
||
|
||
### 3.4 `settings` — peer registry
|
||
|
||
```sql
|
||
CREATE TABLE IF NOT EXISTS peers (
|
||
peer_id TEXT PRIMARY KEY, -- sha256 of node keypair public part
|
||
addr TEXT, -- e.g. http://192.168.1.20:8931
|
||
name TEXT,
|
||
last_handshake TEXT,
|
||
cursor INTEGER, -- max hcl consumed from this peer (pull)
|
||
created_at TEXT DEFAULT (datetime('now'))
|
||
);
|
||
```
|
||
|
||
## 4. Determinism Fixes (prerequisite)
|
||
|
||
Land before gossip so observed facts have stable identity:
|
||
|
||
**F1 — Canonical source paths.** Ingesters must not fingerprint machine-local
|
||
absolute paths. Change:
|
||
- `SessionDiffIngester`: fingerprint on content + session_id (already content-based; verified stable — no change).
|
||
- `LogIngester` (`log.go:72`): release-content-independent → fingerprint on (base filename, line count, extracted facts) or content hash of full text; `SourcePath` stored as filename only.
|
||
- `GitIngester` (`git.go:144`): fingerprint on `"git:repo:"+remote_url+branch` instead of local path.
|
||
- `ObsidianIngester` (`obsidian.go:88-91`): fingerprint on note content + relative vault path (already has `relPath`); drop the absolute path from meta.
|
||
|
||
**F2 — HLC for all "when" storage.** Replace fact-time `time.Now()` with the
|
||
node's HLC tick in `RecordObservation`, entry updates, thread updates,
|
||
`UpsertSession`. `signalTime`/`gitStatusTime` "fall back to now" become *explicit
|
||
unknown* (`""`/NULL) so downstream can rebase rather than fabricate a time.
|
||
|
||
**F3 — Stable TF-IDF sort.** In `tfidf.go`, sort by `(score desc, term asc)` so
|
||
ties are deterministic. Apply in `TopTerms` and cluster naming.
|
||
|
||
**F4 — Locator IDs.** Composite `(node_id, hcl)` for cross-node uniqueness. Within
|
||
a single node the rowid remains monotonic, so the current dedup query
|
||
(`ORDER BY id DESC`) can be re-expressed as `ORDER BY hcl DESC`.
|
||
|
||
## 5. Gossip Protocol
|
||
|
||
### 5.1 Transport
|
||
|
||
Plain HTTP/JSON on a per-node advertized address (default port `8931`). Peers
|
||
are seeded from a static list (`KNOX_PEERS`), then the swarm discovers itself:
|
||
each node advertises its known peer addresses in `/v1/ping`, and every sweep
|
||
enqueues newly-learned nodes for direct contact (membership gossip — no relay of
|
||
observations). mDNS/rendezvous is future work.
|
||
|
||
Endpoints:
|
||
|
||
```
|
||
GET /v1/ping → { node_id, name, max_hcl, peers: [{peer_id, addr, name}] }
|
||
GET /v1/log?after=<seq>&node=<id> → { cursor, rows: [observation...] } # pull
|
||
POST /v1/obs/batch → body: [observation...]; reply: { accepted n, conflict n } # push
|
||
GET /v1/diff → divergence summary (M4)
|
||
```
|
||
|
||
### 5.2 Knowledge exchange
|
||
|
||
- **Knowledge vector:** each node tracks `peer_id → max_hcl consumed`. Anti-entropy
|
||
is a pull: periodically (and on handshake) query each peer's `/v1/log?after=...`.
|
||
- **Membership gossip:** `/v1/ping` includes the responding node's known peers
|
||
(`peer_id`, `addr`, `name`). The caller merges them into its `peers` table and
|
||
enqueues their addresses for direct sweeps. A new node therefore joins the
|
||
whole swarm by configuring just one seed peer. Membership flows independently
|
||
of data — a node never relays another's observations, only its address.
|
||
- **Push:** on a new local observation, best-effort `POST /v1/obs/batch` to known
|
||
peers. A node does **not** re-broadcast something it merely received (that peer
|
||
already has it and will pull from its origin) — this is the echo/loop
|
||
suppression: ownership by `node_id`.
|
||
- **Handshake:** on discovery, `GET /v1/ping`, then a full pull from the peer's
|
||
current cursor (i.e., since `hcl=0` won't happen; use peer's `max_hcl` as
|
||
"I have everything you have" anchor only if the peer trimmed history).
|
||
|
||
### 5.3 Merge rules
|
||
|
||
- **Observations:** append-only, idempotent via `(node_id, hcl)` PK. `INSERT OR
|
||
IGNORE`. Conflicts are impossible by construction (a given node's HCL is
|
||
strictly monotonic).
|
||
- **Entries/threads/topics/sessions:** *not merged.* They are rebuilt from the
|
||
shared observation log by `knox reconcile` (§6). This keeps the log as the only
|
||
replicated state and sidesteps merge-order dependence (D/F/G).
|
||
- **Threads authored by humans** (edited titles, motivations, notes): merged by
|
||
`cluster_key` with LWW on thread HCL. Editing a thread bumps its HCL. `thread_notes`
|
||
are append-only and replicated as observations-like rows if needed (deferred).
|
||
|
||
### 5.4 Tombstones & deletion
|
||
|
||
Observations are immutable — no deletion. For threads, closing sets
|
||
`resolved_at` + bumps HCL; a tombstoned thread is conveyed by its LWW update, not
|
||
a delete. No hard deletes except operator-initiated local cleanup.
|
||
|
||
### 5.5 Failure & partitions
|
||
|
||
No quorum, no leader. Writes never block on peers. A partitioned node keeps
|
||
accepting local observations with its own HCL; on recovery, pull reconciles
|
||
(§6). The only requirement is that each node's `node_id` is unique and its HCL
|
||
monotonic *locally* — cross-node the HCL only orders causally-related rows.
|
||
|
||
### 5.6 Limits
|
||
|
||
- Observations are small structured rows; batch sizes keep LAN-friendly (e.g.,
|
||
1000/batch). No sharding, no streaming — non-goals.
|
||
- The full log is the limit of what a peer will pull; no compaction in M3 (see
|
||
Future Work).
|
||
|
||
## 6. Reconcile Engine
|
||
|
||
`knox reconcile [--dry-run]`:
|
||
|
||
1. Read all observations ordered by `(node_id, hcl)`.
|
||
2. Rebuild `entries` from scratch: for each fingerprint, fold observations in
|
||
HCL order (deterministic by construction) → titles/summaries/extents.
|
||
3. Recompute `ref_count`/`last_seen` as pure local derivations.
|
||
4. Re-run the intact AutoThreader idempotently via `cluster_key`
|
||
(ON CONFLICT DO UPDATE) so threads match — including ones a peer created.
|
||
5. Reports drift (`n entries would change`, `n threads would be added`) when
|
||
`--dry-run`.
|
||
|
||
`knox gossip diff <peer>` (M4) writes the set of fingerprints only the peer has,
|
||
to preview what reconcile would adopt. Reconcile runs automatically after a pull;
|
||
it is cheap because observations are small and the log is the single source of
|
||
truth.
|
||
|
||
## 7. Operational Constraints
|
||
|
||
- **Same-host (WAL):** unchanged — multiple knox processes share one file.
|
||
- **Multi-host:** each host has its own `index.db`; gossip replicates the
|
||
*observation log only*.
|
||
- **Local caches** (`ref_count`, `last_seen`, `received_at`) never leave the node.
|
||
- **Backups:** backing up the observations log of any one node is a full backup
|
||
(everything else is derived).
|
||
- **Node identity:** `node_id` from a persistent generated keypair, stored in
|
||
`settings`. Rotating it orphans old rows (acceptable; document it).
|
||
|
||
## 8. Milestones
|
||
|
||
**M1 — Determinism fixes (prereq).** DONE. F1 (canonical fingerprints), F2 (HLC
|
||
in all "when" columns), F3 (TF-IDF tie-break), F4 (locator IDs + `ORDER BY hcl`).
|
||
Verify: two fresh DBs ingesting the same real content produce identical
|
||
observation hashes and identical threads (minus node_id).
|
||
|
||
**M2 — Composite PK + reconcile.** DONE. HCL backfilled from local rowid;
|
||
`(node_id, hcl)` locator UNIQUE index; threads carry `cluster_key` (partial
|
||
UNIQUE index) making auto-creation idempotent; `knox reconcile` rebuilds
|
||
`entries` from the log and re-links threads. Verify: reconcile is idempotent (0
|
||
create/0 link on re-run); a log-only DB reconstructs `entries` and thread
|
||
`cluster_key`s bit-identical to the original.
|
||
|
||
**M3 — Peer protocol.** DONE. `peers` table; HTTP API (`GET /v1/ping` with
|
||
knowledge vector, `GET /v1/log?node=&after=` pull with cursor paging,
|
||
`POST /v1/obs/batch` push); `watch` daemon serves its log and runs a periodic
|
||
anti-entropy sweep (`KNOX_PEERS`, `KNOX_PEER_ADDR`), reconcile-on-pull; echo
|
||
suppression by `node_id` ownership; `knox gossip status`. Verify: two nodes
|
||
converge to identical logs after a bidirectional sweep (integration test
|
||
`internal/watch/gossip_test.go`); second sweep is idempotent; `time.Now()`-free
|
||
fact paths confirmed by grep.
|
||
|
||
**M4 — Ops & UX.** DONE. `knox gossip` subcommand: `status`, `diff <peer-url>`
|
||
(observation fingerprint divergence + tombstoned auto-thread detection), and
|
||
`sync` (one-shot sweep + reconcile). `/v1/diff` endpoint serves each node's
|
||
knowledge summary. Gossip server starts immediately (before seed) so peers can
|
||
reach a booting node. Config via env `KNOX_PEER_ADDR` / `KNOX_PEERS`.
|
||
Verified e2e: two daemons, `diff` previewed 1655 peer-only fingerprints, `sync`
|
||
pulled all and converged B to 1655 observations.
|
||
|
||
## 9. Future Work (explicitly out of M1–M4)
|
||
|
||
- mDNS / rendezvous peer discovery.
|
||
- Log compaction / pruning with tombstones for cutoff.
|
||
- Replicating `thread_notes` and multi-writer thread editing as CRDT lww-reg
|
||
pairs.
|
||
- Authentication/TLS for non-trusted networks (M3 assumes trusted LAN).
|
||
|
||
## 10. Open Questions
|
||
|
||
- Should `skills-catalog`/`observations` from `filesystem` sources ever sync, or
|
||
stay per-machine by design? (Default: sync all observation sources.)
|
||
- When a human edits a thread on two nodes concurrently, accept last-write-wins?
|
||
(Proposal: yes for M3, revisit with CRDT lww later.)
|
||
- Do we need a heartbeat/tombstone for *peer* removal, or is a soft "unreachable"
|
||
state enough? |