Skip to content

ADR-0078 — Aggregate file ownership: index.md, log.md, overview.md (v1.7.0)

  • Status: Accepted
  • Date: 2026-07-14
  • Supersedes: the per-ingest overview regeneration introduced in F3 (ADR-0073 §overview)
  • Amends: ADR-0076 (block-based pipeline) — removes the post-write hook that called _update_overview from every pipeline execution path
  • Invariants touched: I1, I5, I7, I8
  • Reference: docs/reference/LLMWIKI-CORE-LOGIC-v0.6.3.md §1.8 (llm_wiki aggregate ownership as authoritative behavior spec)

Context

Synapse has three "aggregate" files written by code rather than the LLM:

File Role
wiki/index.md K3 catalogue: all live pages grouped by type (wikilinks)
wiki/log.md K4 append-only ingest history
wiki/overview.md F3 narrative summary of the vault, generated by the ingest provider

Prior to this ADR: - index.md was regenerated by app/wiki/index.py::update_index on every write — correct. - log.md appended one entry per ingest using a date-grouped bullet format (## YYYY-MM-DD then bullet items) — divergent from the reference. - overview.md was regenerated by a blocking _update_overview() call from all three pipeline execution paths (delegated ≈ line 555, block ≈ line 680, orchestrated ≈ line 729 in ADR-0076 pipeline.py) — divergent from the reference and operationally expensive.

§1.8 of the reference (nashsu/llm_wiki v0.6.3) establishes the authoritative ownership contract:

index.md: code-owned; appends - [[slug]] — title lines into a "## Recently Updated" section; deduplicated by slug; capped at 200 entries; updated after every write.

log.md: code-owned; one ## [YYYY-MM-DD] ingest | Title heading per ingest event; no date-grouping wrapper; no bullet list.

overview.md: NEVER regenerated by the pipeline; exists only when explicitly triggered by the operator.

The three pre-ADR-0078 call sites were the largest performance divergence from the reference: on a 10-file ingest batch, _update_overview was called up to 10 times (once per source), each requiring a provider round-trip and a vault read.

Decision

1. index.md — hybrid model (Synapse extends the reference)

Synapse keeps its K3 per-type catalogue (not present in the reference — it is a documented Synapse extension from sprint 1) and adds the code-owned "## Recently Updated" section from §1.8:

  • Section is rendered before the per-type catalogue (prominent placement).
  • Entries: - [[<slug>]] — <title> one per line.
  • Source: a single SELECT (title, page_type, file_path, updated_at) … ORDER BY updated_at DESC query; catalogue re-sorts in Python by title within sections (no second DB round-trip).
  • Deduplication by filename stem (slug); first/most-recently-updated occurrence wins.
  • Cap: 200 entries (constant _RECENTLY_UPDATED_CAP).
  • Exclusions: aggregate types (index, log, overview) and paths starting with raw/.
  • Implementation: app/wiki/index.py::_build_recently_updated() + update_index().

2. log.md — llm_wiki §1.8 heading format

append_log() in app/ingest/orchestrator.py is changed to emit one self-contained Markdown heading per ingest event:

## [YYYY-MM-DD] ingest | <Title>

Rules: - No date-grouping wrapper (## YYYY-MM-DD as a section) — every entry is its own heading. - Verb: ingest (was "indexed" — the reference uses the operation noun). - Subject: page title when available, else the relative source path. - Entries are appended in chronological order (file opened in append mode).

3. overview.md — manual-op only (no pipeline ownership)

All three _update_overview(…) call sites are removed from app/ingest/pipeline.py. The function itself remains in app/ingest/orchestrator.py (as the full implementation, not a stub) so existing test monkeypatches continue to work.

A new public surface is introduced: - app/ops/overview.py::regenerate_overview(analysis, origin_source) — thin delegation to orch._update_overview; the single sanctioned way to regenerate overview.md. - POST /ops/overview/regenerate (router: app/routers/ops_overview.py) — HTTP endpoint returning {"status": "regenerated" | "degraded"}. Degrade-safe (never 5xx). Idempotent.

Callers of regenerate_overview: - The POST /ops/overview/regenerate endpoint (manual operator trigger). - Future scheduled-ops (if added).

The pipeline no longer calls it. wiki/overview.md is therefore absent in a fresh vault until the operator explicitly triggers regeneration.

Consequences

Positive

  • Performance. Overview generation is no longer a blocking step on every ingest. A 10-file batch runs 10× faster (provider round-trips reduced from N to 0).
  • Reference parity. §1.8 ownership contract is now satisfied for all three files.
  • Bounded by design (I7). Overview regeneration is exactly ONE provider call per operator request (bounded at the trigger site, not inside the pipeline loop).
  • Obsidian-valid (I5). The "## Recently Updated" section uses [[wikilinks]] which Obsidian resolves correctly. Frontmatter structure of all three files is unchanged.

Negative / trade-offs

  • overview.md is not auto-maintained. Operators must call POST /ops/overview/regenerate or schedule it to keep the overview current. A UI button is recommended as follow-up.
  • Breaking change in log.md format. Existing log.md files written before this ADR use the date-grouped bullet format. They remain valid Markdown and Obsidian will render them; new entries use the heading format and mix cleanly (headings are self-contained).

Implementation notes

  • app/wiki/index.py: new _build_recently_updated() helper; update_index() now selects 4 columns (adds updated_at); _build_index_content() renders "## Recently Updated" section if the list is non-empty.
  • app/ingest/orchestrator.py::append_log(): format changed from bullet to heading. _update_overview() retained as full implementation (not removed).
  • app/ingest/pipeline.py: three orch._update_overview(…) call sites replaced by ADR-0078 ownership comments. No other pipeline logic changes.
  • app/ops/overview.py: new module; thin delegation using a lazy import of orchestrator to avoid circular imports.
  • app/routers/ops_overview.py: new router; registered in app/main.py.
  • Tests: tests/test_index_recently_updated.py (11 cases), tests/test_overview_not_touched_by_ingest.py (5 cases); existing tests/test_index_md.py updated for 4-tuple query and catalogue-scoped dedup assertion; tests/test_ingest_incremental.py and tests/test_wsg_automations_verification 2.py updated for the new log.md heading format.

Refinement (v1.7.0): queue-drain batch regeneration

Amended status of overview.md ownership: refined from "manual-only" to "once per queue-drain batch".

The original Decision §3 established overview.md as strictly manual-op, removing all pipeline call sites. In practice, the operator-facing POST /ops/overview/regenerate endpoint was rarely triggered and overview.md remained a stub across all real-world vaults after a fresh ingest. The reference implementation (nashsu/llm_wiki) regenerates the overview after every full queue-drain — the completion of the autoIngest event, analogous to Synapse's queue emptying.

Refinement decision (commit 446e66f, v1.7.0):

regenerate_overview() is now called once via the on_drained callback wired in backend/app/main.py, invoked when the ingest queue drains to zero after a batch.

Rules: - Called once per drain event, not per source. A 10-file batch produces one overview call. - Uses provider.complete() (not provider.chat()) — single-turn, no agent loop (ADR-0084). The switch from chat() to complete() is required for correct operation with the CLI provider (CliAgentProvider.chat() creates a full agent loop that hangs on seams without MCP write tools). - Degrade-safe: if the provider call fails or times out (120 s wall-clock, I7), the existing overview.md is retained unchanged. No 5xx is ever raised. - The POST /ops/overview/regenerate endpoint (added in original §3) remains and is the sanctioned on-demand path; the queue-drain call supplements it — it does not replace it.

Rationale for the three options considered:

Option Assessment
Per-source (pre-ADR-0078 behaviour) Starved entity/concept extraction; largest performance bottleneck; up to N round-trips per batch. Removed in original ADR-0078. ✗
Manual-only (original Decision §3) No provider overhead during ingest. In practice overview.md always absent — operators do not trigger the endpoint. ✗
Once per queue-drain (this refinement) Overview reflects the state of the vault after each batch. Bounded at one call per drain event. Reference-parity. ✔

Sequence: the queue-drain section of docs/sequences/ingest-blocks.mmd shows this flow.