Skip to content

stream.confidence

stream.confidence

Advisory confidence scoring for extracted claims.

Confidence is ADVISORY under the auto-publish policy: it is recorded/flagged, not a gate. The seam is the Driver's optional score primitive (supports_logprobs). A logprob-capable backend (e.g. GptDriver against a local vLLM open-weight model) yields calibrated probabilities; backends without logprobs degrade gracefully to a sampling proxy or can be routed to a scorer that does support them.

This module is deliberately store-agnostic: it returns Score objects. Writing those onto Note.grounding and surfacing them in tables.py cells lives in the zettelkasten layer and is wired separately.

pick_scorer 

pick_scorer(primary: Driver, fallback: Driver | None = None) -> Driver | None

Return a logprob-capable driver, preferring primary.

Falls back to fallback when the primary cannot produce logprobs. Returns None when neither can (caller should skip or use the sampling proxy).

Source code in stream/confidence.py 
22
23
24
25
26
27
28
29
30
31
32
def pick_scorer(primary: Driver, fallback: Driver | None = None) -> Driver | None:
    """Return a logprob-capable driver, preferring ``primary``.

    Falls back to ``fallback`` when the primary cannot produce logprobs. Returns
    ``None`` when neither can (caller should skip or use the sampling proxy).
    """
    if getattr(primary, "supports_logprobs", False):
        return primary
    if fallback is not None and getattr(fallback, "supports_logprobs", False):
        return fallback
    return None

score_claim 

score_claim(claim: str, *, driver: Driver, evidence: str = '', fallback: Driver | None = None, model: str = '') -> Score

Score how well evidence supports claim (advisory).

Uses a logprob-capable driver when available (method "logprob"). When none is available it degrades to a single sampled yes/no judgment (method "sampling", confidence=None) so callers always get a Score.

Source code in stream/confidence.py 
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
def score_claim(
    claim: str,
    *,
    driver: Driver,
    evidence: str = "",
    fallback: Driver | None = None,
    model: str = "",
) -> Score:
    """Score how well ``evidence`` supports ``claim`` (advisory).

    Uses a logprob-capable driver when available (method ``"logprob"``). When
    none is available it degrades to a single sampled yes/no judgment
    (method ``"sampling"``, ``confidence=None``) so callers always get a Score.
    """
    prompt = (
        "Does the evidence support the claim? Answer with a single word: "
        "yes or no.\n\n"
        f"CLAIM: {claim}\n"
        f"EVIDENCE: {evidence or '(none provided)'}"
    )
    scorer = pick_scorer(driver, fallback)
    if scorer is not None:
        score = scorer.score(prompt, choices=["yes", "no"], model=model)
        # Normalize so confidence reflects support (probability of "yes").
        if score.value.strip().lower().startswith("no") and score.confidence is not None:
            score.confidence = 1.0 - score.confidence
        return score

    # Sampling proxy: no logprobs anywhere. A bare judgment, confidence unknown.
    text = driver.run_agent(
        role="checker",
        persona="You are a strict grounding judge. Reply yes or no only.",
        prompt=prompt,
        tools=[],
        model=model,
    ).text
    supported = text.strip().lower().startswith("y")
    return Score(value="yes" if supported else "no", confidence=None, method="sampling")

is_low_confidence 

is_low_confidence(score: Score, threshold: float = _LOW_CONFIDENCE) -> bool

Flag a score as low-confidence (advisory only; never gates publish).

Source code in stream/confidence.py 
75
76
77
def is_low_confidence(score: Score, threshold: float = _LOW_CONFIDENCE) -> bool:
    """Flag a score as low-confidence (advisory only; never gates publish)."""
    return score.confidence is not None and score.confidence < threshold