Skip to content

Run experiments: artifacts, backups, and cloud reruns

The memory-artifacts MCP server captures provenance for experiments (code SHA, data hashes, environment, command), backs up large files to a DVC/S3 remote, and reruns experiments in a clean sandbox at the original commit. For the concepts behind it, see Experiments & AWS; this page is the task recipes.

Prerequisites

Install the extra (pip install "angelo[artifacts]") and configure the S3 remote once per repo. The tools live in the separate memory-artifacts MCP server — enable it in your editor's MCP settings if the experiment / artifact tools aren't visible.

Mental model

  • Experiments are series of dated runs sharing an experiment family id.
  • Each run has a manifest: git keeps the manifest (code SHA, data hashes, env, command); DVC/S3 keeps the heavy bytes.
  • A rerun checks out the original commit, pulls inputs by content hash, runs the manifest command, and writes a new dated run with a replication verdict (exact / divergent). It never overwrites the original.

Almost every tool defaults to dry_run=true — run it once to see the plan, then pass dry_run=false to act.

Back up a single file

The one-step path when you just want a large local file safely on the remote — the file stays where it is:

artifact(action="backup", path="results/model.ckpt")

This runs dvc add, pushes to the remote, and verifies the pointer in one call. Commit the resulting .dvc pointer file to git; the bytes live on S3.

Capture a provenance manifest

Record everything needed to reproduce a run. artifacts_json is a comma-separated list of the run's output paths:

experiment(
  action="run",
  experiment_id="bench-20260701",
  command="python bench.py --config configs/a.yaml",
  artifacts_json="results/metrics.json, results/model.ckpt",
  execute=false,          # true to actually run the command now
  dry_run=false,
)

The manifest pins the current code SHA, hashes of the declared artifacts, the environment, and the exact command. Pass include_dirty_diff=true (with an optional dirty_note) if the working tree isn't clean and you want the diff captured too.

Ship and verify a manifest

experiment(action="push_manifest", experiment_id="bench-20260701")     # upload artifacts
experiment(action="verify_manifest", experiment_id="bench-20260701")   # check pointers/files

To produce a human-readable S3 catalog alongside DVC's hash-addressed cache:

experiment(action="export_catalog", experiment_id="bench-20260701")

Rerun from a manifest

Rerun in a clean sandbox at the original commit — locally or on a runner:

experiment(action="rerun", experiment_id="bench-20260701", where="local")   # run here
experiment(action="rerun", experiment_id="bench-20260701", where="cloud")   # enqueue for a runner
  • Rerun outputs land in .memory/artifacts/experiments/<family>/runs/<stamp>/ and never overwrite originals or your local files.
  • A cloud rerun enqueues a job on the DVC remote's jobs queue; an angelo-runner daemon on any machine with bucket credentials executes it.

Poll a queued/running cloud rerun:

experiment(action="get_job", job_id="<id-from-rerun>")

It returns the job state plus a tail of the log (tail_chars, default 8000).

Quick reference

Goal Call
Back up a local file (keep it in place) artifact(action="backup", path=…)
Capture a provenance manifest experiment(action="run", …)
Upload a manifest's artifacts experiment(action="push_manifest", experiment_id=…)
Check a manifest's pointers/files experiment(action="verify_manifest", experiment_id=…)
Export a readable S3 catalog experiment(action="export_catalog", experiment_id=…)
Rerun in a clean sandbox experiment(action="rerun", where="local"\|"cloud")
Poll a cloud rerun experiment(action="get_job", job_id=…)

Prefer dry_run=true first for anything that touches the remote.

Semantically related entries from the memory graph.