zettelkasten.dashboard.backend.routes._common

zettelkasten.dashboard.backend.routes._common

Shared imports, module state, and helpers for the dashboard routes package.

These were the top matter + cross-domain helpers of the former flat routes.py. Every name here is re-exported (see __all__ at the end) so the per-domain sub-modules can from ._common import * and resolve exactly the globals they did when everything lived in one module.

warmup_state

warmup_state() -> dict

Snapshot of background warm-up progress for /api/health.

ready flips True once :func:run_warmup finishes (success OR best-effort failure — search simply warms lazily thereafter, as it did before eager warm-up existed). error carries a repr of any unexpected failure.

Source code in zettelkasten/dashboard/backend/routes/_common.py

def warmup_state() -> dict:
    """Snapshot of background warm-up progress for ``/api/health``.

    ``ready`` flips True once :func:`run_warmup` finishes (success OR best-effort
    failure — search simply warms lazily thereafter, as it did before eager
    warm-up existed). ``error`` carries a repr of any unexpected failure.
    """
    return {"ready": _warmup_done.is_set(), "error": _warmup_error}

warm_model

warm_model() -> None

Prefetch + load the embedding model (one warm-up unit).

The embedder lazily downloads (~120MB, mirror/HuggingFace) and loads on its first use. Paying that inside the first semantic-search or /frame request can exceed an upstream gateway timeout (the dashboard's dev proxy returns 504), so it is primed here. Best-effort: any failure just leaves search to warm lazily as before.

Source code in zettelkasten/dashboard/backend/routes/_common.py

def warm_model() -> None:
    """Prefetch + load the embedding model (one warm-up unit).

    The embedder lazily downloads (~120MB, mirror/HuggingFace) and loads on its
    first use. Paying that inside the first semantic-search or ``/frame`` request
    can exceed an upstream gateway timeout (the dashboard's dev proxy returns
    504), so it is primed here. Best-effort: any failure just leaves search to
    warm lazily as before.
    """
    try:
        from memory.embedders import prefetch_model

        ok, msg = prefetch_model()
        logger.info("Embedding model prefetch: %s (%s)", "ok" if ok else "unavailable", msg)
    except Exception as exc:
        logger.warning("Embedding model prefetch failed: %s", exc)

warm_box

warm_box(name: str) -> None

Build + embed one note graph's kglite index (one warm-up unit).

Touches the graph so its kglite index is built and embedded now; a trivial search forces the index to materialize. Best-effort per box.

Source code in zettelkasten/dashboard/backend/routes/_common.py

def warm_box(name: str) -> None:
    """Build + embed one note graph's kglite index (one warm-up unit).

    Touches the graph so its kglite index is built and embedded now; a trivial
    search forces the index to materialize. Best-effort per box.
    """
    try:
        zg = _get_graph(name)
        if zg.notes and getattr(zg, "embeddings", None) is not None:
            zg.embeddings.search("warmup", top_k=1)
    except Exception as exc:
        logger.debug("Embedding warmup skipped for box '%s': %s", name, exc)

warm_embeddings

warm_embeddings() -> None

Prime the embedding model and every per-box index, synchronously.

Retained as the single-call form for direct/test callers. The dashboard backend does NOT call this on the request-serving path; it uses the chunked :func:run_warmup so warm-up never blocks the health probe. Both share the same units, so behaviour is identical aside from scheduling.

Source code in zettelkasten/dashboard/backend/routes/_common.py

def warm_embeddings() -> None:
    """Prime the embedding model and every per-box index, synchronously.

    Retained as the single-call form for direct/test callers. The dashboard
    backend does NOT call this on the request-serving path; it uses the chunked
    :func:`run_warmup` so warm-up never blocks the health probe. Both share the
    same units, so behaviour is identical aside from scheduling.
    """
    warm_model()
    # ``note_graph_names`` already includes ``_cross`` when present.
    for name in note_graph_names():
        warm_box(name)

run_warmup async

run_warmup() -> None

Prime the embedder model; build per-box indexes LAZILY (on demand).

Startup warm-up now does the minimum: it loads the shared embedder model once (cheap, process-cached) and returns. It deliberately does NOT walk every note-graph building its kglite index. The old build-everything loop serialized ~one native box build per request token — and main.lifespan caps anyio's thread limiter at a SINGLE token (native-heap safety, see :data:embeddings._NATIVE_LOCK) — so on a large store (thousands of notes / hundreds of boxes) warm-up monopolized that lone token for minutes and starved the static UI, which needs the same token to serve. Most of those boxes are never opened in a session, so the work was largely wasted latency.

Instead we rely on the self-healing on-demand path already wired into every embedding route: the first time a semantic feature touches a cold box, :func:_maybe_rewarm schedules a single background build (:func:schedule_rewarm), the request returns embeddings_status="warming" immediately, and the next poll returns real results — and the box stays cached on disk thereafter. The net effect is "project-scoped" warm-up: opening a project warms only ITS boxes, a few seconds after the graph is already interactive, rather than blocking the whole UI on the entire library up front.

Priming the model here (not per-box) keeps the one shared, non-_NATIVE_LOCK native load (Model2VecAdapter.load) off the first on-demand build's latency while costing only a single token for ~a second, once.

Set ZETTEL_EAGER_WARMUP=1 to restore the old build-every-box behaviour (useful for a dedicated/headless instance where up-front cost is acceptable and every-box readiness is wanted). Records completion (and any unexpected error) in the module warm-up state so /api/health can report readiness.

Source code in zettelkasten/dashboard/backend/routes/_common.py

async def run_warmup() -> None:
    """Prime the embedder model; build per-box indexes LAZILY (on demand).

    Startup warm-up now does the minimum: it loads the shared embedder model once
    (cheap, process-cached) and returns. It deliberately does NOT walk every
    note-graph building its kglite index. The old build-everything loop serialized
    ~one native box build per request token — and ``main.lifespan`` caps anyio's
    thread limiter at a SINGLE token (native-heap safety, see
    :data:`embeddings._NATIVE_LOCK`) — so on a large store (thousands of notes /
    hundreds of boxes) warm-up monopolized that lone token for minutes and starved
    the static UI, which needs the same token to serve. Most of those boxes are
    never opened in a session, so the work was largely wasted latency.

    Instead we rely on the self-healing on-demand path already wired into every
    embedding route: the first time a semantic feature touches a cold box,
    :func:`_maybe_rewarm` schedules a single background build (:func:`schedule_rewarm`),
    the request returns ``embeddings_status="warming"`` immediately, and the next
    poll returns real results — and the box stays cached on disk thereafter. The
    net effect is "project-scoped" warm-up: opening a project warms only ITS boxes,
    a few seconds after the graph is already interactive, rather than blocking the
    whole UI on the entire library up front.

    Priming the model here (not per-box) keeps the one shared, non-``_NATIVE_LOCK``
    native load (``Model2VecAdapter.load``) off the first on-demand build's latency
    while costing only a single token for ~a second, once.

    Set ``ZETTEL_EAGER_WARMUP=1`` to restore the old build-every-box behaviour
    (useful for a dedicated/headless instance where up-front cost is acceptable and
    every-box readiness is wanted). Records completion (and any unexpected error)
    in the module warm-up state so ``/api/health`` can report readiness.
    """
    import anyio

    error: str | None = None
    try:
        await anyio.to_thread.run_sync(warm_model)
        if os.environ.get("ZETTEL_EAGER_WARMUP", "").strip().lower() in ("1", "true", "yes"):
            # Opt-in legacy path: build every local box up front. Each unit holds
            # the sole thread token for its duration; the ``await`` between units
            # lets a queued request acquire it between boxes so nothing native
            # overlaps.
            names = await anyio.to_thread.run_sync(note_graph_names)
            for name in names:
                await anyio.to_thread.run_sync(warm_box, name)
    except Exception as exc:  # best-effort: never crash startup over warm-up
        error = repr(exc)
        logger.warning("Embedding warm-up failed: %s", exc)
    finally:
        # Reached on success, best-effort failure, AND cancellation (a
        # BaseException that bypasses the except above) — health must always
        # stop reporting "warming" once the task is no longer running.
        _mark_warmup_done(error)

schedule_rewarm

schedule_rewarm(name: str) -> bool

Idempotently schedule a background kglite rebuild for a cold box.

Returns True when a build was scheduled, False when one is already in flight for name (dedup) or scheduling failed. Never blocks and never touches native code on the caller's thread: the actual build runs on the single-worker background executor, whose native ops serialize behind _NATIVE_LOCK exactly like request-thread native ops do.

Source code in zettelkasten/dashboard/backend/routes/_common.py

def schedule_rewarm(name: str) -> bool:
    """Idempotently schedule a background kglite rebuild for a cold box.

    Returns ``True`` when a build was scheduled, ``False`` when one is already in
    flight for ``name`` (dedup) or scheduling failed. Never blocks and never
    touches native code on the caller's thread: the actual build runs on the
    single-worker background executor, whose native ops serialize behind
    ``_NATIVE_LOCK`` exactly like request-thread native ops do.
    """
    with _rewarm_lock:
        if name in _rewarm_in_progress:
            return False
        _rewarm_in_progress.add(name)
    try:
        _rewarm_executor.submit(_rewarm_worker, name)
    except Exception as exc:  # executor shut down / cannot accept work
        # Clear the marker so a later request can retry rather than leaving the
        # box wedged on "warming" because a build was reserved but never ran.
        with _rewarm_lock:
            _rewarm_in_progress.discard(name)
        logger.debug("Could not schedule re-warm for box '%s': %s", name, exc)
        return False
    return True

owner_routes

owner_routes(router, suffix, endpoints, *, methods)

Register a project+graph twin pair through a single call.

endpoints maps owner_type ("project"/"graph") to the thin delegator that forwards to the shared owner_type-parametrized helper (_promote, _verify_org, _spine_group_matrix, _remine_* ...). The twin paths differ only in the /projects vs /graphs prefix, so routing both declarations through here defines each pair's registration once while emitting byte-for-byte the same (method, path) pair, the same response model (inferred from the delegator's return annotation, exactly as the @router decorator did), and the same parameter handling. Order is project-then-graph, matching the original module's registration order.

Source code in zettelkasten/dashboard/backend/routes/_common.py

def owner_routes(router, suffix, endpoints, *, methods):
    """Register a project+graph twin pair through a single call.

    ``endpoints`` maps ``owner_type`` ("project"/"graph") to the thin delegator
    that forwards to the shared ``owner_type``-parametrized helper (``_promote``,
    ``_verify_org``, ``_spine_group_matrix``, ``_remine_*`` ...). The twin paths differ
    only in the ``/projects`` vs ``/graphs`` prefix, so routing both declarations
    through here defines each pair's registration once while emitting byte-for-byte
    the same ``(method, path)`` pair, the same response model (inferred from the
    delegator's return annotation, exactly as the ``@router`` decorator did), and
    the same parameter handling. Order is project-then-graph, matching the
    original module's registration order.
    """
    for owner_type in ("project", "graph"):
        endpoint = endpoints.get(owner_type)
        if endpoint is None:
            continue
        router.add_api_route(
            f"{_OWNER_PREFIX[owner_type]}/{{name}}{suffix}",
            endpoint,
            methods=list(methods),
        )