Skip to content

stream.scoring

stream.scoring

Post-extraction confidence scoring pass.

After a grounded-extraction run completes, score how well each freshly extracted claim/finding is supported by its verbatim quote evidence and persist the result onto the note's grounding (advisory governance — never a publish gate). The score then lights up the dashboard note panel and the synthesis-matrix cells via zettelkasten.tables.

The pass needs a logprob-capable scoring driver (e.g. a GptDriver pointed at a local vLLM open-weight model). Without one it degrades gracefully: claims with no numeric confidence are simply left unscored, and the matrix/UI show nothing rather than a fabricated number.

score_sources 

score_sources(source_graphs: list[str], *, driver: Driver, model: str = '', fallback: Driver | None = None, rescore: bool = False) -> dict[str, Any]

Score + persist advisory confidence for claim/finding notes.

Iterates each source graph's claim/finding notes, resolves the supporting quote (its evidence), asks the driver to judge support, and writes the resulting probability onto the note via :func:zettelkasten.server.set_note_confidence.

Already-scored notes are skipped unless rescore is set. Notes the scorer could not produce a numeric confidence for (no logprobs anywhere) are left untouched. Returns a summary {scored, skipped, graphs, ...}.

Source code in stream/scoring.py 
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
def score_sources(
    source_graphs: list[str],
    *,
    driver: Driver,
    model: str = "",
    fallback: Driver | None = None,
    rescore: bool = False,
) -> dict[str, Any]:
    """Score + persist advisory confidence for claim/finding notes.

    Iterates each source graph's claim/finding notes, resolves the supporting
    quote (its evidence), asks the driver to judge support, and writes the
    resulting probability onto the note via
    :func:`zettelkasten.server.set_note_confidence`.

    Already-scored notes are skipped unless ``rescore`` is set. Notes the scorer
    could not produce a numeric confidence for (no logprobs anywhere) are left
    untouched. Returns a summary ``{scored, skipped, graphs, ...}``.
    """
    from zettelkasten import server as zk
    from zettelkasten.tables import _find_quote_evidence

    scored = 0
    skipped = 0
    no_confidence = 0
    for graph in source_graphs:
        try:
            zg = zk._get_graph(graph)
        except Exception as e:  # pragma: no cover - defensive
            logger.warning("score_sources: cannot load graph %s: %s", graph, e)
            continue
        # Snapshot values() — set_note_confidence mutates+saves as we go.
        for note in list(zg.notes.values()):
            if note.type not in _CLAIM_TYPES:
                continue
            g = note.grounding if isinstance(note.grounding, dict) else None
            if not rescore and g and g.get("score") is not None:
                skipped += 1
                continue
            claim_text = (note.body or note.title or "").strip()
            if not claim_text:
                continue
            ev = _find_quote_evidence(zg, note.id)
            evidence = ev["quote"] if ev else ""
            try:
                sc = score_claim(
                    claim_text,
                    driver=driver,
                    evidence=evidence,
                    fallback=fallback,
                    model=model,
                )
            except Exception as e:  # pragma: no cover - driver/runtime errors
                logger.warning("score_claim failed for %s/%s: %s", graph, note.id, e)
                continue
            if sc.confidence is None:
                # Sampling proxy (no logprobs): no numeric confidence to persist.
                no_confidence += 1
                continue
            zk.set_note_confidence(graph, note.id, sc.confidence, method=sc.method)
            scored += 1

    result: dict[str, Any] = {
        "scored": scored,
        "skipped": skipped,
        "graphs": len(source_graphs),
    }
    if no_confidence:
        result["no_confidence"] = no_confidence
        if scored == 0:
            result["note"] = (
                "scoring driver produced no logprobs; no confidence persisted. "
                "Point score_driver at a logprob-capable backend (e.g. gpt/vLLM)."
            )
    return result

score_run_prep 

score_run_prep(prep: dict[str, Any], *, driver: Driver, **kwargs: Any) -> dict[str, Any]

Convenience: score the source graphs recorded in a Run's prep.

Source code in stream/scoring.py 
107
108
109
110
111
112
def score_run_prep(prep: dict[str, Any], *, driver: Driver, **kwargs: Any) -> dict[str, Any]:
    """Convenience: score the source graphs recorded in a Run's ``prep``."""
    sources = [s.get("name", "") for s in (prep.get("sources") or []) if s.get("name")]
    if not sources:
        return {"scored": 0, "skipped": 0, "graphs": 0}
    return score_sources(sources, driver=driver, **kwargs)