zettelkasten.bundle_builder

zettelkasten.bundle_builder

Background, process-isolated builder for per-project view bundles.

This is Stage C of the load re-architecture: it moves the expensive native work (markdown parse + embedding + t-SNE + clustering) that produces a project's :mod:zettelkasten.bundle OFF the web server and into a ProcessPoolExecutor.

Why a process pool and not a thread pool: kglite (a native extension with process-global state) and the scientific stack (sklearn/numpy/BLAS) corrupt each other's heap if run concurrently in the SAME process — the reason the dashboard caps its own sync threadpool to a single token and serializes native ops under embeddings._NATIVE_LOCK. Separate processes each have their OWN native heap, so several bundle builds run truly in parallel across cores with no shared-heap risk, and none of them touch the web server's heap at all. The web server stays a lightweight reader: on a bundle miss it schedules a build here and keeps serving its existing (live) response; the next poll, once the bundle lands, is an instant signature-gated read.

The scheduler is idempotent and deduplicated (mirrors _common.schedule_rewarm): a burst of polls for the same project spawns at most one in-flight build, and the in-progress marker is cleared in a done callback so a failed build is retryable by a later request rather than wedged.

Disable entirely with ZETTEL_BUNDLE_POOL=0 (the routes then just serve their live path with no background warming). Worker count defaults to a small fraction of the machine and is overridable with ZETTEL_BUNDLE_WORKERS.

schedule_bundle_build

schedule_bundle_build(project: str) -> bool

Idempotently schedule a background bundle build for project.

Returns True when a build was scheduled, False when one is already in flight for this project (dedup), the pool is disabled/unavailable, or scheduling failed. Never blocks and never runs native code on the caller's thread — the actual build runs in the process pool.

Source code in zettelkasten/bundle_builder.py

def schedule_bundle_build(project: str) -> bool:
    """Idempotently schedule a background bundle build for ``project``.

    Returns ``True`` when a build was scheduled, ``False`` when one is already in
    flight for this project (dedup), the pool is disabled/unavailable, or
    scheduling failed. Never blocks and never runs native code on the caller's
    thread — the actual build runs in the process pool.
    """
    if not project or not _pool_enabled():
        return False
    executor = _get_executor()
    if executor is None:
        return False
    with _lock:
        if project in _in_progress:
            return False
        _in_progress.add(project)
    try:
        fut = executor.submit(_build_bundle_task, project)
    except Exception as exc:  # pool shut down / cannot accept work
        with _lock:
            _in_progress.discard(project)
        logger.debug("Could not schedule bundle build for %r: %s", project, exc)
        return False
    fut.add_done_callback(_make_done_callback(project))
    return True

build_in_progress

build_in_progress(project: str) -> bool

True when a background bundle build for project is currently in flight.

Source code in zettelkasten/bundle_builder.py

def build_in_progress(project: str) -> bool:
    """True when a background bundle build for ``project`` is currently in flight."""
    with _lock:
        return project in _in_progress

enable_write_prewarm

enable_write_prewarm(enabled: bool = True) -> None

Turn write-time bundle pre-warm on (called by a long-lived server startup).

Source code in zettelkasten/bundle_builder.py

def enable_write_prewarm(enabled: bool = True) -> None:
    """Turn write-time bundle pre-warm on (called by a long-lived server startup)."""
    global _prewarm_enabled
    _prewarm_enabled = enabled

note_written

note_written(box_name: str) -> None

Cheap per-write signal (fired from ZettelGraph.save_note/delete_note).

No-op unless write-time pre-warm was enabled AND the build pool is enabled. Records the touched box and (re)arms a debounce timer so a burst of writes coalesces into one rebuild per affected project once the boxes go quiet. Never blocks and never runs native code on the caller's thread.

Source code in zettelkasten/bundle_builder.py

def note_written(box_name: str) -> None:
    """Cheap per-write signal (fired from ``ZettelGraph.save_note``/``delete_note``).

    No-op unless write-time pre-warm was enabled AND the build pool is enabled.
    Records the touched box and (re)arms a debounce timer so a burst of writes
    coalesces into one rebuild per affected project once the boxes go quiet. Never
    blocks and never runs native code on the caller's thread.
    """
    global _prewarm_timer
    if not _prewarm_enabled or not box_name or not _pool_enabled():
        return
    with _prewarm_lock:
        _pending_boxes.add(box_name)
        if _prewarm_timer is not None:
            _prewarm_timer.cancel()
        timer = threading.Timer(_prewarm_debounce_sec(), _fire_prewarm)
        timer.daemon = True
        _prewarm_timer = timer
        timer.start()

shutdown

shutdown(wait: bool = False) -> None

Tear down the build pool (best-effort). Safe to call when never started.

Also DISABLES write-prewarm: a note_written that lands after (or racing) teardown — a late write during app shutdown, or the in-process chat agent finishing a save — would otherwise pass the _prewarm_enabled gate, arm a fresh timer, and on fire re-create a ProcessPoolExecutor that nothing tears down again (orphaned workers). Clearing the flag under the same lock closes that window; a server that restarts calls :func:enable_write_prewarm again.

Source code in zettelkasten/bundle_builder.py

def shutdown(wait: bool = False) -> None:
    """Tear down the build pool (best-effort). Safe to call when never started.

    Also DISABLES write-prewarm: a ``note_written`` that lands after (or racing)
    teardown — a late write during app shutdown, or the in-process chat agent
    finishing a save — would otherwise pass the ``_prewarm_enabled`` gate, arm a
    fresh timer, and on fire re-create a ``ProcessPoolExecutor`` that nothing tears
    down again (orphaned workers). Clearing the flag under the same lock closes
    that window; a server that restarts calls :func:`enable_write_prewarm` again.
    """
    global _executor, _prewarm_timer, _prewarm_enabled
    with _prewarm_lock:
        _prewarm_enabled = False
        if _prewarm_timer is not None:
            _prewarm_timer.cancel()
            _prewarm_timer = None
        _pending_boxes.clear()
    with _lock:
        ex = _executor
        _executor = None
        _in_progress.clear()
    if ex is not None:
        try:
            ex.shutdown(wait=wait, cancel_futures=True)
        except Exception:  # pragma: no cover
            logger.debug("Bundle pool shutdown error", exc_info=True)