Skip to content

Ground a claim in data

Ground a claim in a re-computable statistic instead of a verbatim quote: pin a dataset, write a method: data citation, and let the auditor verify it by re-running the computation. This is the data half of the evidence contract; for the concepts behind it (the three grounding modalities, epistemic_status, and the "verify, never compose" boundary) see Grounded extraction.

Before you begin

You need the zettelkasten bundle enabled and a source graph (box) to write into. The claim grounds on a dataset note in that box (or a linked one). A descriptive statistic (mean, quantile, sum, …) verifies anywhere via the built-in expr evaluator with no extra setup; a fitted quantity needs a host-registered derivation (see step 2).

Throughout, the worked example is a sizing claim about an investor's position concentration: "Buffett's top-5 positions average 68% of the equity book."

1. Snapshot the derived frame into a dataset box

Ground on the small derived frame the statistic runs over — not a multi-GB raw panel. Add it as a dataset note, then snapshot it to pin a content hash so the citation grounds on a frozen frame:

dataset(action="add", graph="buffett", title="Top-5 concentration by quarter",
        path="top5_concentration.csv", format="csv",
        body="Quarterly top-5 position weight, derived from 13F filings.")

dataset(action="snapshot", graph="buffett", note_id="<dataset note id>")

snapshot re-reads the current bytes and writes their content_hash into the note, returning the new hash (and whether it changed). Copy that hash — it is the dataset_hash your citation pins. Confirm the columns the derivation needs with dataset(action="get", ...) (metadata) or dataset(action="values", ...) (rows).

2. Choose a derivation: the expr hatch or a registered one

The derivation is the computation that turns the frame into a number.

  • expr hatch (no setup, verifies anywhere). A whitelisted expression: mean:col, median:col, sum:col, std:col, count:col, min:col, max:col, or quantile:col:<q> (with q in [0, 1]). For the example, the derivation id is mean:top5_concentration. Use this for any descriptive statistic — it re-verifies in any repo, so the claim is portable.
  • Registered derivation (for a fitted quantity). Host code contributed via register_derivation, keyed by (id, version) — the same trust boundary as a custom agent in agents.yaml. Use this when the number is a fit (an inverse-optimization, a model calibration):
from zettelkasten.data_grounding import register_derivation, DerivationResult

@register_derivation(id="mydomain.concentration.mean_top5", version=3,
                     fit=False, deterministic=True)
def mean_top5(table, selection, params) -> DerivationResult:
    ...
    return DerivationResult(value=0.68)

fit=False marks a measured derivation; fit=True marks an inferred one. A deterministic=False derivation must pin an RNG seed in the citation's params, or verification fails with nondeterministic_unpinned. A registered derivation is loaded at server startup, so restart the zettelkasten MCP server after adding one.

3. Write the claim with a method: data grounding

Add the claim note with the citation and its epistemic_status. A data-grounded claim needs no supporting quote — its evidence is the citation:

note(action="add", graph="buffett",
     title="Top-5 positions average 68% of the book",
     type="finding",
     body="Across quarters, the top-5 positions average 68% of the equity book — a high-conviction, concentrated sizing posture.",
     tags=["sizing"],
     epistemic_status="measured",
     grounding={
       "method": "data",
       "dataset_note": "<dataset note id>",
       "dataset_hash": "<content_hash from step 1>",
       "selection": {"where": "quarter >= 2005-01-01"},
       "derivation_id": "mean:top5_concentration",
       "value": 0.68,
       "tolerance": 1.0e-6,
       "confidence": 0.9
     })
  • selection is a deterministic row/column filter (where is a conjunction of simple col <op> literal comparisons; columns restricts columns). Omit it for the whole frame.
  • For a registered derivation, use its id and add derivation_version (and any params, e.g. {"seed": 7}); for an expr, omit the version.
  • epistemic_status is measured for a descriptive statistic, inferred for a fitted one. Set it explicitly.
  • Do not set verified / verified_at / verified_in yourself — the verifier stamps those on a clean re-run. confidence (and any validation / method_meta blob) is stored and displayed but never computed with.

4. Verify by recompute

Re-run the citation on demand. This re-resolves the pinned dataset, re-runs the derivation, and compares the result to the asserted value within tolerance:

note(action="verify_grounding", graph="buffett", note_id="<claim id>")

The structured result reports the outcome. On a clean reproduce the note is stamped verified / verified_at / verified_in and saved. Otherwise the result names the failure tier:

  • Hard violation (a real defect that blocks coverage): dataset_hash_mismatch (the pinned data changed), value_not_reproduced (the re-run missed value beyond tolerance), nondeterministic_unpinned (a stochastic derivation with no pinned seed), or validation_malformed.
  • Soft unavailable (informational, not a defect): derivation_unavailable (the derivation isn't registered here) or dataset_unavailable (the dataset isn't resolvable here). The claim keeps any prior verified stamp — an expr claim verifies anywhere, so a soft outcome on one signals a genuinely missing dataset.

5. Read the coverage epistemic mix

The coverage matrix counts a verified data claim as covered, exactly like a quote-backed claim, and reports a per-dimension mix of how each dimension is grounded:

source(action="coverage", source="buffett", schema="<schema name>")

Each dimension row carries an epistemic count — for example sizing: {grounded: 0, measured: 1, inferred: 0} — so you can tell at a glance whether a dimension is backed by verbatim source (grounded), directly-observed statistics (measured), or fitted quantities (inferred). A hard data violation drops that claim from the mix and surfaces it as unreproduced_data_claims in source(action="verify", source="buffett").

Quick reference

Goal Call
Pin a derived frame dataset(action="snapshot", graph=…, note_id=…)
Inspect a dataset's rows/columns dataset(action="values" \| "get", graph=…, note_id=…)
Register a fitted derivation register_derivation(id=…, version=…, fit=True)
Write a data-grounded claim note(action="add", …, grounding={method: "data", …}, epistemic_status=…)
Verify by recompute note(action="verify_grounding", graph=…, note_id=…)
Read the epistemic mix source(action="coverage", source=…, schema=…)

Descriptive statistics travel with the claim (portable, re-verifiable anywhere); fitted quantities verify where their derivation is installed and are trusted by their verified_in provenance elsewhere.

Semantically related entries from the memory graph.