Skip to content

Experiments and AWS

How angelo makes an experiment reproducible on a machine that never touched the original run — and how a second machine reruns it without ever talking to the first one directly.

The memory dashboard Experiments tab: a list of experiment runs with pass/fail status, tags, and DVC-backed manifests, split from lighter tests

The Experiments tab — provenance manifests joined with their memory entries (with a pass/fail status and a separate Tests sub-tab).

Why it exists

A bare result file is not an experiment. To trust an output you need to know the code that produced it, the inputs it consumed, and the command that wired them together — and you need that record to survive long after the original working tree is gone. Angelo also runs across more than one machine: a laptop where work is scoped and a compute box (typically AWS) that has the cycles and the credentials. Those two machines have no direct link — no SSH session, no RPC, no always-on socket between them.

Why split provenance across three layers

Code, large bytes, and the recipe that joins them have different lifecycles. Code is small and belongs in git history. Artifacts are large, regenerable, and content-addressable — they belong in object storage, not git. The recipe (which commit, which command, which input hashes) is a tiny immutable record that pins the other two together. Keeping each in the store that fits it is what makes a run replayable on a fresh machine instead of merely archived.

How it works

The three-layer model

Every experiment is recorded as three cooperating layers:

  1. Git commit — code, .dvc pointer files (a path plus its content md5 and size), memory entries, and the manifest itself. This is the small, versioned, diff-able layer. A manifest captures git_commit = HEAD at run time (see build_experiment_manifest in memory/artifacts.py).
  2. DVC / S3 content-addressed artifacts — the actual bytes. DVC stores each file under its md5 in the S3 bucket configured as the DVC remote; git keeps only the pointer. Because storage is keyed by content hash, identical bytes are stored once and any copy can be integrity-checked against its hash.
  3. Manifest — the provenance record at .memory/artifacts/manifests/<id>.yaml: schema version, id, git_commit, command, cwd, a dirty-tree flag/diff, a redacted environment summary, the artifact list (each with its md5), an artifact_state (local_only → queued → remote_verified, or failed), and the remote it was pushed to. Secret-looking keys are scrubbed before the manifest is committed.

A rerun never needs the original machine: git_commit + command + cwd + content-hashed inputs is a complete, replayable recipe. The manifest even records each input's md5 so an input can be materialized from DVC even when its pointer file was committed after the manifest's git_commit — a real lag, since pointers are usually committed just after a run.

The cloud-rerun architecture

The two machines collaborate through shared storage only. The S3 bucket carries both the artifact bytes (DVC cache) and a job queue under <DVC remote>/jobs — small JSON files moved between pending/, running/, done/, and failed/ prefixes, with logs under jobs/logs/ (see memory/jobs.py). The queue base is resolved from ANGELO_JOBS_URI or, by default, the DVC remote URL plus /jobs.

Any machine Git remote S3 bucket AWS runner git pull write job JSON poll + claim commit + push run manifest pull inputs push outputs experiment rerun where = cloud manifests + .dvc pointers + memory jobs/ queue pending → running → done DVC cache bytes keyed by md5 claim_next pending → running git-worktree sandbox at original commit rerun + replication run command in recorded cwd
Two machines, no direct link. They collaborate only through shared storage: the S3 jobs queue and DVC cache (amber = byte transfers), plus the git remote for manifests. Results flow home after a git pull.

Cloud mode (experiment(action='rerun', where='cloud')) resolves the target manifest, builds a job spec, and enqueues it. The angelo-runner daemon (memory/runner.py) polls the queue on the AWS box, claims the oldest pending job (claim_next atomically moves it to running/), git pulls so the referenced manifest is local, then calls rerun_experiment. That:

  • creates a git-worktree sandbox detached at the original commit,
  • reconstructs and dvc pulls inputs by content hash,
  • runs the manifest command in the recorded cwd,
  • copies outputs into a per-run folder (.memory/artifacts/experiments/<family>/runs/<stamp>/), DVC-adds and pushes them,
  • writes a new dated run manifest carrying rerun_of, executor, and a replication check.

Results flow home the same decoupled way they came: artifact bytes via S3, and the new run manifest via git commit + git push to the shared remote, so every other machine sees the run after a git pull. The runner streams its log back to jobs/logs/, so the originating machine can poll progress with experiment(action='get_job', job_id) without any direct connection.

The replication check

The rerun compares each regenerated output's md5 against the original output's md5 (read from the pointer as it stood at the original commit) and grades the run:

  • exact — every output hash matches; bit-for-bit reproduction.
  • divergent — at least one output differs (or is missing). Often legitimate: if the original run was dirty (uncommitted changes), the sandbox reran the committed code, so a caveat is recorded explaining the gap.
  • unknown — outputs were produced but no original hash was available to compare against.

What's available

Concern Where it lives
Manifest build / read / verify, sandbox rerun, replication memory/artifacts.py
S3 job queue (state prefixes, claim/heartbeat/logs) memory/jobs.py
angelo-runner daemon (poll → rerun → publish) memory/runner.py
MCP tools: artifact, pipeline, experiment memory/artifact_tools.py

These tools live in the separate memory-artifacts MCP server, so the block can be toggled independently of the core memory server. Using the remote requires the [artifacts] extra and a configured S3 DVC remote (artifact(action='configure_remote')). For the step-by-step task — capturing a manifest, backing up a file, kicking off a rerun — see Run experiments.

Design notes

The decisions behind this, drawn as a slice of the memory tree.

Semantically related entries from the memory graph.