memory.reranking

memory.reranking

Blended recall-then-rerank scoring core (store-agnostic).

Borrows FinMem's idea of fusing three signals — similarity, importance (graph centrality) and access-recency — into ranking via a recall-then-rerank stage:

1. A candidate set is recalled by SIMILARITY (the gate — done upstream by the embedding search; this module takes the recalled set as input).
2. The candidates are RERANKED by fusing similarity + importance + recency with weighted Reciprocal Rank Fusion (:func:memory.ranking.rrf_fuse).
3. The fused order is DIVERSIFIED with Maximal Marginal Relevance (MMR).

Non-punitive invariant. Importance and recency are boosts within the already-relevant set. A note that lacks a signal (never accessed, low centrality) simply does not appear in that channel — it is NEVER pushed down for the absence. Concretely: with w_importance == w_recency == 0 and mmr_lambda == 1.0, :func:rerank returns the candidates in pure similarity order. This is the guardrail that keeps the reranker from degenerating into a decay/forgetting mechanism.

This module is a pure scoring core: it holds no MCP wiring and reads no environment or config files. Callers construct a :class:RerankConfig (or take :func:default_config) and pass it in.

It lives in memory (the base layer) so BOTH the memory tree and the zettelkasten can use it without violating the one-way dependency direction (zettelkasten may import memory; memory never imports zettelkasten). zettelkasten.reranking re-exports these names for backward compatibility and adds the syllabus-backed source-importance channel.

RerankConfig dataclass

Tunable weights and knobs for :func:rerank.

Attributes:

Name	Type	Description
w_sim	float	Weight of the similarity channel. Keep > 0 — it is the gate that carries every recalled candidate into the fusion.
w_importance	float	Weight of the graph-centrality channel (0 disables it).
w_recency	float	Weight of the access-recency channel (0 disables it).
w_calendar_recency	float	Weight of the CALENDAR-recency channel (0 disables it — the default). Distinct from w_recency (which is session/access recency): this ranks by the source material's chronological age, so newer material outranks older at equal similarity. It is non-punitive: an item that carries no date is simply absent from the channel and never penalized.
calendar_half_life	float	Half-life in DAYS for the calendar-recency decay (~90 = one quarter). Consumed by the WIRING that builds the calendar map, not by :func:rerank itself; carried here so the whole policy lives in one object.
mmr_lambda	float	MMR trade-off in [0, 1]. 1.0 = pure fused relevance (no diversification); lower values trade relevance for diversity against already-selected notes. 1.0 preserves the non-punitive invariant.
overfetch	int	Recall-stage over-fetch multiplier (candidates to recall before reranking). Consumed by the WIRING that builds the candidate set, not by :func:rerank itself; carried here so the whole policy lives in one object.
rrf_k	int | None	RRF damping constant. None defers to :func:memory.ranking.rrf_fuse's default (ANGELO_RRF_K or 60).
use_source_importance	bool	Whether to fold the (expensive) syllabus source-work importance channel in. OFF by default; the wiring opts in and supplies the map. :func:rerank itself is agnostic — it only sees the importance_map it is handed.

Source code in memory/reranking.py

@dataclass
class RerankConfig:
    """Tunable weights and knobs for :func:`rerank`.

    Attributes:
        w_sim: Weight of the similarity channel. Keep > 0 — it is the gate that
            carries every recalled candidate into the fusion.
        w_importance: Weight of the graph-centrality channel (0 disables it).
        w_recency: Weight of the access-recency channel (0 disables it).
        w_calendar_recency: Weight of the CALENDAR-recency channel (0 disables
            it — the default). Distinct from ``w_recency`` (which is
            session/access recency): this ranks by the source material's
            chronological age, so newer material outranks older at equal
            similarity. It is non-punitive: an item that carries no date is
            simply absent from the channel and never penalized.
        calendar_half_life: Half-life in DAYS for the calendar-recency decay
            (~90 = one quarter). Consumed by the WIRING that builds the calendar
            map, not by :func:`rerank` itself; carried here so the whole policy
            lives in one object.
        mmr_lambda: MMR trade-off in [0, 1]. 1.0 = pure fused relevance (no
            diversification); lower values trade relevance for diversity against
            already-selected notes. 1.0 preserves the non-punitive invariant.
        overfetch: Recall-stage over-fetch multiplier (candidates to recall
            before reranking). Consumed by the WIRING that builds the candidate
            set, not by :func:`rerank` itself; carried here so the whole policy
            lives in one object.
        rrf_k: RRF damping constant. ``None`` defers to
            :func:`memory.ranking.rrf_fuse`'s default (``ANGELO_RRF_K`` or 60).
        use_source_importance: Whether to fold the (expensive) syllabus
            source-work importance channel in. OFF by default; the wiring opts in
            and supplies the map. :func:`rerank` itself is agnostic — it only
            sees the ``importance_map`` it is handed.
    """

    w_sim: float = 1.0
    w_importance: float = 0.5
    w_recency: float = 0.5
    w_calendar_recency: float = 0.0
    calendar_half_life: float = 90.0
    mmr_lambda: float = 0.7
    overfetch: int = 3
    rrf_k: int | None = None
    use_source_importance: bool = False

Candidate

Bases: NamedTuple

A similarity-recalled candidate.

Matches the hit shape produced by zettelkasten.framing.project_query ((note_id, cosine, source_graph, keyword_tier)). Because it is a NamedTuple it unpacks positionally, so :func:rerank also accepts plain tuples of that shape interchangeably.

A candidate's IDENTITY is the composite (source_graph, note_id), not the bare note_id: zettel ids are not namespaced by box, so the same id can name distinct notes in different source graphs. Single-graph callers (e.g. the memory tree) may pass a constant source_graph (such as "").

Source code in memory/reranking.py

class Candidate(NamedTuple):
    """A similarity-recalled candidate.

    Matches the hit shape produced by ``zettelkasten.framing.project_query``
    (``(note_id, cosine, source_graph, keyword_tier)``). Because it is a
    ``NamedTuple`` it unpacks positionally, so :func:`rerank` also accepts plain
    tuples of that shape interchangeably.

    A candidate's IDENTITY is the composite ``(source_graph, note_id)``, not the
    bare ``note_id``: zettel ids are not namespaced by box, so the same id can
    name distinct notes in different source graphs. Single-graph callers (e.g.
    the memory tree) may pass a constant ``source_graph`` (such as ``""``).
    """

    note_id: str
    similarity: float
    source_graph: str
    keyword_tier: int | None = None

default_config

default_config() -> RerankConfig

Return a :class:RerankConfig with sensible defaults.

Source code in memory/reranking.py

def default_config() -> RerankConfig:
    """Return a :class:`RerankConfig` with sensible defaults."""
    return RerankConfig()

note_importance

note_importance(zg: Any, *, method: str = 'degree') -> dict[str, float]

Per-note centrality for a graph, in [0, 1], max-normalized.

v1 (method="degree") is the LOG of undirected degree (outgoing note.links + incoming backlinks), max-normalized to [0, 1].

NOTE on the log: fusion is RANK-based (:func:memory.ranking.rrf_fuse scores an id by its RANK within each channel, not by the channel's magnitude), and log1p is monotonic in degree — so the log does NOT change the importance channel's ordering versus raw degree. Anti-entrenchment comes instead from the importance WEIGHT (w_importance) and the MMR diversity pass — never from this log.

method is the seam for a future directed-PageRank replacement over note.links; only "degree" is implemented today (any other value raises ValueError). Results are cached per graph and invalidated when the graph reloads (see :data:_importance_cache).

zg is duck-typed: any object exposing .notes (a mapping of id -> object with .id and .links where each link has a .target) works, so this serves both the zettelkasten graph and the memory tree graph.

Source code in memory/reranking.py

def note_importance(zg: Any, *, method: str = "degree") -> dict[str, float]:
    """Per-note centrality for a graph, in ``[0, 1]``, max-normalized.

    v1 (``method="degree"``) is the LOG of undirected degree (outgoing
    ``note.links`` + incoming backlinks), max-normalized to ``[0, 1]``.

    NOTE on the log: fusion is RANK-based (:func:`memory.ranking.rrf_fuse` scores
    an id by its RANK within each channel, not by the channel's magnitude), and
    ``log1p`` is monotonic in degree — so the log does NOT change the importance
    channel's ordering versus raw degree. Anti-entrenchment comes instead from
    the importance WEIGHT (``w_importance``) and the MMR diversity pass — never
    from this log.

    ``method`` is the seam for a future directed-PageRank replacement over
    ``note.links``; only ``"degree"`` is implemented today (any other value
    raises ``ValueError``). Results are cached per graph and invalidated when the
    graph reloads (see :data:`_importance_cache`).

    ``zg`` is duck-typed: any object exposing ``.notes`` (a mapping of id ->
    object with ``.id`` and ``.links`` where each link has a ``.target``) works,
    so this serves both the zettelkasten graph and the memory tree graph.
    """
    try:
        signature = _graph_signature(zg, method)
        cached = _importance_cache.get(zg)
        if cached is not None and cached[0] == signature:
            return cached[1]
    except TypeError:
        # A non-weakref-able / unhashable graph stand-in (some tests) simply
        # bypasses caching; correctness is unaffected.
        signature = None

    if method == "degree":
        degree = _undirected_degree(zg)
    else:
        raise ValueError(f"unknown importance method: {method!r}")

    scores = _max_normalize({nid: math.log1p(d) for nid, d in degree.items()})

    if signature is not None:
        try:
            _importance_cache[zg] = (signature, scores)
        except TypeError:  # pragma: no cover - defensive
            pass
    return scores

rerank

rerank(candidates: 'list[Any]', *, importance_map: dict[str, float], recency_map: dict[str, float], calendar_map: dict[str, float] | None = None, cfg: RerankConfig | None = None, get_vector: 'Callable[[str, str], list[float] | None] | None' = None, limit: 'int | None' = None) -> list[dict]

Rerank similarity-recalled candidates by fusing sim + importance + recency.

Parameters:

Name	Type	Description	Default
candidates	'list[Any]'	The similarity-recalled set, each a :class:Candidate or a (note_id, cosine, source_graph[, keyword_tier]) tuple. Duplicates by the COMPOSITE (source_graph, note_id) identity are collapsed (first occurrence wins); the same id in two different graphs stays distinct.	required
importance_map	dict[str, float]	importance keyed by (source_graph, note_id). A bare note_id key is accepted as a fallback for single-graph callers. Notes absent from it simply don't appear in the importance channel — never penalized.	required
recency_map	dict[str, float]	recency keyed by (source_graph, note_id), with the same bare note_id fallback. Absent = no recency channel, never penalized.	required
calendar_map	dict[str, float] | None	CALENDAR-recency keyed by (source_graph, note_id), same bare-note_id fallback. Higher = chronologically newer material. Absent = absent from the channel, never a penalty. Weighted by cfg.w_calendar_recency (0 = off).	None
cfg	RerankConfig | None	Weights/knobs; :func:default_config when None.	None
get_vector	'Callable[[str, str], list[float] | None] | None'	Optional (source_graph, note_id) -> vector | None used for the MMR diversity pass. When None (or mmr_lambda >= 1.0) MMR is skipped and the fused relevance order is returned as-is.	None
limit	'int | None'	Optional cap on MMR-selected notes (see :func:_mmr_order).	None

Returns:

Type	Description
list[dict]	Final-best-first list of per-candidate breakdown dicts:
list[dict]	``{id, source_graph, sim, importance, recency, calendar, fused,
list[dict]	keyword_tier}``.

Non-punitive invariant: with w_importance == w_recency == w_calendar_recency == 0 and mmr_lambda == 1.0 the boost channels contribute nothing and MMR is skipped, so the output is in pure similarity order.

Source code in memory/reranking.py

def rerank(
    candidates: "list[Any]",
    *,
    importance_map: dict[str, float],
    recency_map: dict[str, float],
    calendar_map: dict[str, float] | None = None,
    cfg: RerankConfig | None = None,
    get_vector: "Callable[[str, str], list[float] | None] | None" = None,
    limit: "int | None" = None,
) -> list[dict]:
    """Rerank similarity-recalled candidates by fusing sim + importance + recency.

    Args:
        candidates: The similarity-recalled set, each a :class:`Candidate` or a
            ``(note_id, cosine, source_graph[, keyword_tier])`` tuple. Duplicates
            by the COMPOSITE ``(source_graph, note_id)`` identity are collapsed
            (first occurrence wins); the same id in two different graphs stays
            distinct.
        importance_map: importance keyed by ``(source_graph, note_id)``. A bare
            ``note_id`` key is accepted as a fallback for single-graph callers.
            Notes absent from it simply don't appear in the importance channel —
            never penalized.
        recency_map: recency keyed by ``(source_graph, note_id)``, with the same
            bare ``note_id`` fallback. Absent = no recency channel, never
            penalized.
        calendar_map: CALENDAR-recency keyed by ``(source_graph, note_id)``, same
            bare-``note_id`` fallback. Higher = chronologically newer material.
            Absent = absent from the channel, never a penalty. Weighted by
            ``cfg.w_calendar_recency`` (0 = off).
        cfg: Weights/knobs; :func:`default_config` when ``None``.
        get_vector: Optional ``(source_graph, note_id) -> vector | None`` used for
            the MMR diversity pass. When ``None`` (or ``mmr_lambda >= 1.0``) MMR
            is skipped and the fused relevance order is returned as-is.
        limit: Optional cap on MMR-selected notes (see :func:`_mmr_order`).

    Returns:
        Final-best-first list of per-candidate breakdown dicts:
        ``{id, source_graph, sim, importance, recency, calendar, fused,
        keyword_tier}``.

    Non-punitive invariant: with ``w_importance == w_recency ==
    w_calendar_recency == 0`` and ``mmr_lambda == 1.0`` the boost channels
    contribute nothing and MMR is skipped, so the output is in pure similarity
    order.
    """
    cfg = cfg or default_config()
    if not candidates:
        return []

    # Normalize + dedupe on the COMPOSITE identity ``(source_graph, note_id)``,
    # preserving first-seen order.
    cos: dict[tuple, float] = {}
    sg_map: dict[tuple, str] = {}
    nid_map: dict[tuple, str] = {}
    tier_map: dict[tuple, "int | None"] = {}
    keys: list[tuple] = []
    for c in candidates:
        nid, similarity, source_graph, tier = _cand_fields(c)
        key = (source_graph, nid)
        if key in cos:
            continue
        keys.append(key)
        cos[key] = similarity
        sg_map[key] = source_graph
        nid_map[key] = nid
        tier_map[key] = tier

    # Resolve a channel value for a composite key: prefer the composite entry,
    # then fall back to a bare-``note_id`` entry so single-graph callers may hand
    # in id-keyed maps unambiguously.
    def _channel(m: dict, key: tuple) -> "float | None":
        if key in m:
            return m[key]
        return m.get(key[1])

    calendar_map = calendar_map or {}
    imp_vals: dict[tuple, "float | None"] = {k: _channel(importance_map, k) for k in keys}
    rec_vals: dict[tuple, "float | None"] = {k: _channel(recency_map, k) for k in keys}
    cal_vals: dict[tuple, "float | None"] = {k: _channel(calendar_map, k) for k in keys}

    # Build the rankings RESTRICTED to the candidate set. Importance, recency and
    # calendar rankings include ONLY candidates that carry that signal, so a note
    # missing from a map is absent from that channel (not ranked last).
    sim_rank = sorted(keys, key=lambda k: (-cos[k], k))
    imp_rank = sorted(
        (k for k in keys if imp_vals[k] is not None),
        key=lambda k: (-imp_vals[k], k),
    )
    rec_rank = sorted(
        (k for k in keys if rec_vals[k] is not None),
        key=lambda k: (-rec_vals[k], k),
    )
    cal_rank = sorted(
        (k for k in keys if cal_vals[k] is not None),
        key=lambda k: (-cal_vals[k], k),
    )

    rankings = [sim_rank, imp_rank, rec_rank, cal_rank]
    weights = [cfg.w_sim, cfg.w_importance, cfg.w_recency, cfg.w_calendar_recency]
    fused = rrf_fuse(rankings, k=cfg.rrf_k, weights=weights)

    # Best single-channel rank over CONTRIBUTING channels, for deterministic
    # tie-breaks (mirrors memory.ranking.rrf_order).
    best_rank: dict[tuple, int] = {}
    for weight, ranking in zip(weights, rankings):
        if weight == 0.0:
            continue
        for rank, key in enumerate(ranking, start=1):
            if rank < best_rank.get(key, _BIG):
                best_rank[key] = rank

    # Order by fused score, then best rank, then similarity, then key. When only
    # the similarity channel is active this reduces to pure similarity order.
    ordered = sorted(
        keys,
        key=lambda k: (-fused.get(k, 0.0), best_rank.get(k, _BIG), -cos[k], k),
    )

    # Diversify the fused order with MMR unless it is switched off. Relevance for
    # MMR is the fused score max-normalized to [0, 1] so it is comparable to the
    # cosine-similarity penalty.
    if get_vector is not None and cfg.mmr_lambda < 1.0:
        order_pos = {k: i for i, k in enumerate(ordered)}
        relevance = _max_normalize({k: fused.get(k, 0.0) for k in ordered})
        vectors: dict[tuple, "list[float] | None"] = {}
        for k in ordered:
            try:
                vec = get_vector(sg_map[k], nid_map[k])
            except Exception:  # pragma: no cover - defensive: never fail rerank
                vec = None
            vectors[k] = list(vec) if vec else None
        ordered = _mmr_order(
            ordered, relevance, vectors, cfg.mmr_lambda, order_pos, limit=limit
        )

    return [
        {
            "id": nid_map[k],
            "source_graph": sg_map[k],
            "sim": cos[k],
            "importance": float(imp_vals[k]) if imp_vals[k] is not None else 0.0,
            "recency": float(rec_vals[k]) if rec_vals[k] is not None else 0.0,
            "calendar": float(cal_vals[k]) if cal_vals[k] is not None else 0.0,
            "fused": fused.get(k, 0.0),
            "keyword_tier": tier_map[k],
        }
        for k in ordered
    ]