zettelkasten.tables_gather

zettelkasten.tables_gather

Deterministic cell-gather / row-scope / routing logic for the matrix engine.

Mechanically split out of tables.py: column-member routing, row + group scope construction (source/tag/note_type/group incl. link & semantic), and the gather_rows / list_row_values entry points.

gather_rows

gather_rows(get_graph: GetGraph, columns: list[dict[str, Any]], *, project: str = '', graph: str = '', row_axis: 'dict[str, Any] | None' = None, graphs_dir: 'Path | None' = None, localize: 'Callable[[str], str] | None' = None, with_material: bool = False, group_fn: 'Callable[[str, str], str] | None' = None, classify_fn: 'Callable[..., Any] | None' = None, matrix_view: 'dict[str, Any] | None' = None, attributions: 'dict[tuple[str, str], str] | None' = None, collision_out: 'dict[str, Any] | None' = None, row_vectors: 'dict[str, list[float]] | None' = None) -> list[dict[str, Any]]

Build one row per row-axis entity with deterministic cells pre-filled.

Pure: no LLM, no writes. The row axis (source default, or tag / note_type) decides how the corpus is partitioned into rows; columns backed by prompt are left as [GAP] here for the injectable EXTRACT pass to fill, every other backing is resolved from the row's scoped notes.

When with_material is set, each row carries a transient material key (a text digest of its scoped notes) for the EXTRACT pass to ground on. The caller MUST strip it before persisting/returning — it is grounding input, not grid data.

classify_fn (default None) wires the re-mining note->dimension classifier through the deterministic fill: when supplied it AUGMENTS each non-prompt cell with agent-inferred members (see :func:_route_members). Left None, no LLM runs and the gathered grid is byte-identical to the deterministic-only output. Build the production classifier via :func:zettelkasten.remine.build_classify_fn.

row_vectors (default None) opts a group/semantic row axis into the HYBRID partition: a {"<graph>::<id>" -> embedding} map that lets embedding similarity propose candidate clusters which the agent then names and refines (see :func:_group_semantic_scopes). It is inert on every other axis and, left None/empty, the semantic axis behaves exactly as the historical pure-agent path — so hybrid grouping is strictly opt-in.

matrix_view (the spine's tree → grid flatten config, §11.3) controls the spine-side membership readback. Its rollup flag decides whether a dimension cell AGGREGATES its component-of subtree. When a cols_level PIVOT is in effect, each column (a structure node at that cut, chosen upstream by :func:build_matrix) reads its cell from the rolled-up index AT THAT PIVOT — so rollup is forced on and the apex stays in the index (for the cols_level == 0 apex column), guaranteeing members below the cut roll up with no silent loss. None (the default) is the historical flat grid — rollup on, apex excluded — so a non-spine table or an org with no matrix_view gathers byte-identically to before V2b.

Source code in zettelkasten/tables_gather.py

def gather_rows(
    get_graph: GetGraph,
    columns: list[dict[str, Any]],
    *,
    project: str = "",
    graph: str = "",
    row_axis: "dict[str, Any] | None" = None,
    graphs_dir: "Path | None" = None,
    localize: "Callable[[str], str] | None" = None,
    with_material: bool = False,
    group_fn: "Callable[[str, str], str] | None" = None,
    classify_fn: "Callable[..., Any] | None" = None,
    matrix_view: "dict[str, Any] | None" = None,
    attributions: "dict[tuple[str, str], str] | None" = None,
    collision_out: "dict[str, Any] | None" = None,
    row_vectors: "dict[str, list[float]] | None" = None,
) -> list[dict[str, Any]]:
    """Build one row per row-axis entity with deterministic cells pre-filled.

    Pure: no LLM, no writes. The row axis (``source`` default, or ``tag`` /
    ``note_type``) decides how the corpus is partitioned into rows; columns backed
    by ``prompt`` are left as ``[GAP]`` here for the injectable EXTRACT pass to
    fill, every other backing is resolved from the row's scoped notes.

    When ``with_material`` is set, each row carries a transient ``material`` key
    (a text digest of its scoped notes) for the EXTRACT pass to ground on. The
    caller MUST strip it before persisting/returning — it is grounding input, not
    grid data.

    ``classify_fn`` (default ``None``) wires the re-mining note->dimension
    classifier through the deterministic fill: when supplied it AUGMENTS each
    non-prompt cell with agent-inferred members (see :func:`_route_members`).
    Left ``None``, no LLM runs and the gathered grid is byte-identical to the
    deterministic-only output. Build the production classifier via
    :func:`zettelkasten.remine.build_classify_fn`.

    ``row_vectors`` (default ``None``) opts a ``group``/``semantic`` row axis into
    the HYBRID partition: a ``{"<graph>::<id>" -> embedding}`` map that lets
    embedding similarity propose candidate clusters which the agent then names and
    refines (see :func:`_group_semantic_scopes`). It is inert on every other axis
    and, left ``None``/empty, the semantic axis behaves exactly as the historical
    pure-agent path — so hybrid grouping is strictly opt-in.

    ``matrix_view`` (the spine's tree → grid flatten config, §11.3) controls the
    spine-side membership readback. Its ``rollup`` flag decides whether a dimension
    cell AGGREGATES its ``component-of`` subtree. When a ``cols_level`` PIVOT is in
    effect, each column (a structure node at that cut, chosen upstream by
    :func:`build_matrix`) reads its cell from the rolled-up index AT THAT PIVOT —
    so ``rollup`` is forced on and the apex stays in the index (for the
    ``cols_level == 0`` apex column), guaranteeing members below the cut roll up
    with no silent loss. ``None`` (the default) is the historical flat grid —
    ``rollup`` on, apex excluded — so a non-spine table or an org with no
    ``matrix_view`` gathers byte-identically to before V2b.
    """
    base = Path(graphs_dir) if graphs_dir is not None else GRAPHS_DIR
    loc = localize or (lambda s: s)
    axis = normalize_row_axis(row_axis)
    # Default (matrix_view=None) → flat grid with rollup on, exactly as before V2b.
    mv = normalize_matrix_view(matrix_view) if matrix_view is not None else None
    rollup = mv["rollup"] if mv is not None else True
    # A column-axis PIVOT (``cols_level`` set) reads each column's cell from the
    # rolled-up index at that cut, so members attached STRICTLY BELOW the pivot
    # roll up into it with NO SILENT LOSS — this forces ``rollup`` on regardless of
    # the stored flag, and (for the ``cols_level == 0`` apex column) keeps the apex
    # in the index. With ``cols_level=None`` both stay at their byte-identical
    # defaults, so a non-pivot/non-spine table reads back exactly as before V2b.
    pivoting = mv is not None and mv["cols_level"] is not None
    if pivoting:
        rollup = True
    exclude_apex = not pivoting
    scopes = _build_row_scopes(
        get_graph, axis, project=project, graph=graph, base=base, loc=loc,
        group_fn=group_fn, attributions=attributions, collision_out=collision_out,
        row_vectors=row_vectors,
    )
    # Memoized spine-membership accessor: a promoted spine's columns route their
    # cell members via the dimension node's outgoing ``spine-member`` edges, read
    # spine-side. Built once per spine graph (keyed by name) and shared across rows
    # so a wide grid loads each spine graph at most once. The configured ``rollup``
    # flag flows through here so the stored ``matrix_view`` actually takes effect.
    _spine_cache: dict[str, dict[str, set[str]]] = {}

    def _spine_members(graph_name: str) -> dict[str, set[str]]:
        idx = _spine_cache.get(graph_name)
        if idx is None:
            idx = _spine_membership_index(
                get_graph, loc, graph_name, rollup=rollup, exclude_apex=exclude_apex
            )
            _spine_cache[graph_name] = idx
        return idx

    rows: list[dict[str, Any]] = []
    for scope in scopes:
        cells: dict[str, Any] = {}
        for col in columns:
            if col["backing"] == "prompt":
                cells[col["key"]] = _gap_cell()
            else:
                cells[col["key"]] = _deterministic_cell(
                    scope, col, spine_members=_spine_members, classify_fn=classify_fn
                )
        row = {
            "id": scope.id,
            "label": scope.label,
            "source_graph": scope.source_graph,
            "cells": cells,
        }
        if with_material:
            row["material"] = _scope_material(scope)
        rows.append(row)
    rows.sort(key=lambda r: r["label"].lower())
    return rows

list_row_values

list_row_values(get_graph: GetGraph, *, kind: str = 'source', ref: str = '', project: str = '', graph: str = '', strategy: str = '', relation: str = '', direction: str = 'outgoing', apex: str = '', instruction: str = '', group_fn: 'Callable[[str, str], str] | None' = None, graphs_dir: 'Path | None' = None, localize: 'Callable[[str], str] | None' = None) -> list[dict[str, Any]]

Enumerate the candidate row values for an axis, with note counts.

Feeds the builder wizard's auto-listed, curatable row picker. source lists the corpus sources (count = notes); tag / note_type list the distinct values across the cross-graph universe (count = notes carrying the value), sorted by frequency so the dominant entities surface first. A group axis delegates to the scope builder so the listed rows are exactly the rows a build would produce (count = notes aggregated into that row); a semantic group needs group_fn (the agent) to enumerate and returns nothing without it.

Source code in zettelkasten/tables_gather.py

def list_row_values(
    get_graph: GetGraph,
    *,
    kind: str = "source",
    ref: str = "",
    project: str = "",
    graph: str = "",
    strategy: str = "",
    relation: str = "",
    direction: str = "outgoing",
    apex: str = "",
    instruction: str = "",
    group_fn: "Callable[[str, str], str] | None" = None,
    graphs_dir: "Path | None" = None,
    localize: "Callable[[str], str] | None" = None,
) -> list[dict[str, Any]]:
    """Enumerate the candidate row values for an axis, with note counts.

    Feeds the builder wizard's auto-listed, curatable row picker. ``source`` lists
    the corpus sources (count = notes); ``tag`` / ``note_type`` list the distinct
    values across the cross-graph universe (count = notes carrying the value),
    sorted by frequency so the dominant entities surface first. A ``group`` axis
    delegates to the scope builder so the listed rows are exactly the rows a build
    would produce (count = notes aggregated into that row); a ``semantic`` group
    needs ``group_fn`` (the agent) to enumerate and returns nothing without it.
    """
    base = Path(graphs_dir) if graphs_dir is not None else GRAPHS_DIR
    loc = localize or (lambda s: s)
    kind = kind if kind in _ROW_AXES else "source"

    if kind == "group":
        axis = normalize_row_axis(
            {
                "kind": "group",
                "ref": ref,
                "strategy": strategy,
                "relation": relation,
                "direction": direction,
                "apex": apex,
                "instruction": instruction,
            }
        )
        # A ``spine`` axis lists the schema's member spines directly (one row per
        # spine), NOT generic link targets — see :func:`_list_spine_row_values`.
        if axis.get("strategy") == "spine":
            return _list_spine_row_values(
                get_graph, axis, project=project, graph=graph, base=base, loc=loc
            )
        scopes = _build_row_scopes(
            get_graph, axis, project=project, graph=graph, base=base,
            loc=loc, group_fn=group_fn,
        )
        out = [
            {"value": s.id, "label": s.label, "count": len(s.notes)} for s in scopes
        ]
        out.sort(key=lambda d: (-d["count"], d["label"].lower()))
        return out

    if kind == "source":
        out: list[dict[str, Any]] = []
        for source in _scope_sources(
            get_graph, project=project, graph=graph, graphs_dir=base, localize=loc
        ):
            try:
                zg = get_graph(loc(source))
            except Exception:
                continue
            meta = load_source_meta(source, graphs_dir=base)
            out.append(
                {"value": source, "label": str(meta.get("title") or source), "count": len(zg.notes)}
            )
        out.sort(key=lambda d: d["label"].lower())
        return out

    if kind == "note":
        out = []
        for name, _zg, note in _universe_pairs(
            get_graph, project=project, graph=graph, base=base, loc=loc
        ):
            if note.type in _NON_VALUE_TYPES:
                continue
            if any(t in _NON_VALUE_TAGS for t in (note.tags or [])):
                continue
            out.append(
                {
                    "value": f"{name}::{note.id}",
                    "label": str(note.title or note.id),
                    "count": len(note.links or []),
                }
            )
        out.sort(key=lambda d: d["label"].lower())
        return out

    counts: dict[str, int] = {}
    for _name, _zg, note in _universe_pairs(
        get_graph, project=project, graph=graph, base=base, loc=loc
    ):
        for value in _value_keys(note, kind):
            counts[value] = counts.get(value, 0) + 1
    out = [{"value": k, "label": k, "count": c} for k, c in counts.items()]
    out.sort(key=lambda d: (-d["count"], d["label"]))
    return out