zettelkasten.organizations_migration

zettelkasten.organizations_migration

Lazy, one-time migration of legacy review tables into organizations.

The review-table MIGRATION logic extracted verbatim from :mod:zettelkasten.organizations (Phase 3 structural decomposition). This is a behavior-preserving relocation: every function keeps its exact signature and body, and :mod:zettelkasten.organizations re-exports each name so all existing zettelkasten.organizations.<name> import (and monkeypatch) paths keep working unchanged.

Shared core helpers (paths, validation, normalization, IO, markers) live in :mod:zettelkasten.organizations; every one is referenced through the _org module alias at CALL TIME (_org.<name>) rather than a frozen from zettelkasten.organizations import <name> value-binding, so a monkeypatch.setattr(organizations, '<name>', ...) on ANY shared helper — not just a curated few — is honored inside the moved functions here. Only symbols DEFINED in this module are referenced directly.

ensure_migrated

ensure_migrated(owner_type: str, owner_name: str, *, graphs_dir: 'Path | None' = None) -> int

Import legacy review tables for this owner into organizations (idempotent).

For every review whose scope matches (owner_type, owner_name) and which has NOT been absorbed yet (per the per-owner marker), each of its persisted tables becomes a lens organization carrying that table's row axis + columns. The review name is then recorded in the marker, so a subsequent call is a no-op and a user deleting a migrated org never resurrects it.

Returns the number of organizations created.

Source code in zettelkasten/organizations_migration.py

def ensure_migrated(
    owner_type: str,
    owner_name: str,
    *,
    graphs_dir: "Path | None" = None,
) -> int:
    """Import legacy review tables for this owner into organizations (idempotent).

    For every review whose scope matches ``(owner_type, owner_name)`` and which has
    NOT been absorbed yet (per the per-owner marker), each of its persisted tables
    becomes a ``lens`` organization carrying that table's row axis + columns. The
    review name is then recorded in the marker, so a subsequent call is a no-op and
    a user deleting a migrated org never resurrects it.

    Returns the number of organizations created.
    """
    base = Path(graphs_dir) if graphs_dir is not None else _org.GRAPHS_DIR
    ot, on = _org._validate_owner(owner_type, owner_name)

    # Cheap pre-check OUTSIDE the lock: which reviews target this owner and are new?
    owner_dir = _org._owner_dir(base, ot, on)
    try:
        review_names = [str(m.get("name") or "").strip() for m in _org.list_reviews(graphs_dir=base)]
    except Exception:
        review_names = []
    candidates = [
        r for r in review_names if r and _review_scope(r, base) == (ot, on)
    ]
    if not candidates:
        return 0
    already = _org._read_marker(owner_dir)
    pending = [r for r in candidates if r not in already]
    # Back-fill of the durable ``migrated`` flag onto any pre-flag legacy org file
    # (cycle-6 P1). Gated on whether an UNSTAMPED legacy migrated org actually
    # remains — NOT a one-shot sentinel — so a legacy org synced/restored in AFTER
    # an earlier back-fill is still caught (the OneDrive cross-machine sync-merge
    # case). Cheap: reads only the marker's provenance-proven legacy org files.
    need_backfill = _needs_backfill(owner_dir, base, already, ot, on)
    if not pending and not need_backfill:
        return 0

    created = 0
    with _org.review_write_lock(_org._owner_lock_name(ot, on), graphs_dir=base):
        # Re-read the marker under the lock to avoid a double-import race.
        marker = _org._read_marker(owner_dir)
        try:
            for review_name in pending:
                if review_name in marker:
                    continue
                tables = _load_legacy_tables(review_name, base)
                for table_id, table in tables.items():
                    if not isinstance(table, dict):
                        continue
                    oid = _migrated_org_id(review_name, str(table_id))
                    # Don't overwrite an org the user already authored under this id.
                    if _org._org_file(base, ot, on, oid).exists():
                        continue
                    org = _org.normalize_organization(
                        {
                            "id": oid,
                            "title": str(table.get("title") or table_id),
                            # Legacy review tables are the legacy DEFAULT VIEWS: they
                            # become live ``lens`` orgs (a read-only projection, no
                            # materialized graph) carrying an EMPTY embedded schema —
                            # they are not spine-backed, so there is no rubric to embed.
                            # ``normalize_organization`` already defaults a missing
                            # ``schema`` to ``{}``; stamping it explicitly makes the
                            # directory-entry contract (state + schema) visible at the
                            # migration site rather than relying on the normalizer.
                            "state": "lens",
                            "schema": {},
                            "row_axis": table.get("row_axis"),
                            "columns": table.get("columns"),
                            "review": review_name,
                            "table_id": str(table_id),
                            # TRUE migration provenance — stamped HERE and nowhere
                            # else. Lets :func:`_org_synthesis_graphs` recognize a
                            # migrated legacy VIEW of a synthesis graph (whose only
                            # binding to G may be a dimension column's ``graph=G``)
                            # WITHOUT also misclassifying an ordinary live-built
                            # review table that merely cross-references G in a
                            # column (``upsert_definition`` never sets this).
                            "migrated": True,
                            # Carry the legacy grid's freshness so a post-migration
                            # peek stays cost-free (grid itself stays in tables.json).
                            "signature": str(table.get("signature") or ""),
                            "generated_at": str(table.get("generated_at") or ""),
                            "created_at": _org._now(),
                            "updated_at": _org._now(),
                        },
                        owner_type=ot,
                        owner_name=on,
                        org_id=oid,
                    )
                    _org._write_org_unlocked(base, org)
                    created += 1
                # Persist the marker the moment a review is FULLY imported, so a
                # crash partway through ``pending`` can't leave a completed
                # review un-marked and thus re-importable — which would resurrect
                # any of its orgs the user deleted in the interim.
                marker.add(review_name)
                _org._write_marker(owner_dir, marker)
            # Durable back-fill of the ``migrated`` flag onto pre-flag legacy org
            # files. Runs against the FULL marker (freshly imported reviews already
            # carry the flag, so the back-fill's guards skip them) whenever an
            # UNSTAMPED legacy migrated org still remains — re-checked under the
            # lock against the fresh marker, so a legacy org that arrives via sync
            # AFTER an earlier back-fill is still stamped (cycle-6 P1; no permanent
            # one-shot gate). The back-fill is idempotent (only stamps files still
            # missing the key), so re-running is safe. A ``_migrated_backfilled.json``
            # breadcrumb records the last pass for observability; it is NO LONGER a
            # gate (a stale sentinel must never suppress a later back-fill).
            if _needs_backfill(owner_dir, base, marker, ot, on):
                _backfill_migrated_flags(owner_dir, ot, on, base, marker)
                _org.atomic_write_text(
                    owner_dir / _org._BACKFILL_FILE,
                    json.dumps({"backfilled_at": _org._now()}, indent=2, ensure_ascii=False),
                )
                _org._schedule_zettel_commit(str((owner_dir / _org._BACKFILL_FILE).resolve()))
        finally:
            # Belt-and-suspenders: persist whatever progress was made and refresh
            # the derived index even if the loop raised mid-flight.
            _org._write_marker(owner_dir, marker)
            if created:
                _org._rebuild_index(owner_dir, ot, on)
    return created

migrate_all

migrate_all(*, graphs_dir: 'Path | None' = None) -> dict[str, int]

Upgrade an entire existing store: import every review's tables into orgs.

Walks all review manifests, derives each one's owner, and runs the per-owner :func:ensure_migrated once per distinct owner. Safe to run repeatedly — it is idempotent (the per-owner marker makes re-runs no-ops) and non-destructive (legacy _reviews/*.tables.json files are read, never modified or removed).

Returns {"<owner_type>/<owner_name>": orgs_created} for each owner touched. Use this to migrate an old directory eagerly; otherwise migration happens lazily the first time an owner is listed.

Source code in zettelkasten/organizations_migration.py

def migrate_all(*, graphs_dir: "Path | None" = None) -> dict[str, int]:
    """Upgrade an entire existing store: import every review's tables into orgs.

    Walks all review manifests, derives each one's owner, and runs the per-owner
    :func:`ensure_migrated` once per distinct owner. Safe to run repeatedly — it
    is idempotent (the per-owner marker makes re-runs no-ops) and non-destructive
    (legacy ``_reviews/*.tables.json`` files are read, never modified or removed).

    Returns ``{"<owner_type>/<owner_name>": orgs_created}`` for each owner touched.
    Use this to migrate an old directory eagerly; otherwise migration happens
    lazily the first time an owner is listed.
    """
    base = Path(graphs_dir) if graphs_dir is not None else _org.GRAPHS_DIR
    try:
        review_names = [str(m.get("name") or "").strip() for m in _org.list_reviews(graphs_dir=base)]
    except Exception:
        review_names = []

    result: dict[str, int] = {}
    seen: set[tuple[str, str]] = set()
    for review_name in review_names:
        if not review_name:
            continue
        scope = _review_scope(review_name, base)
        if scope is None or scope in seen:
            continue
        seen.add(scope)
        ot, on = scope
        result[f"{ot}/{on}"] = ensure_migrated(ot, on, graphs_dir=base)
    return result