Skip to content
noggin

CI budget: caches, xdist and the run's wall time (#219, #220)

Sources: GitHub’s dependency-caching reference (https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching), astral-sh/setup-uv at our pin 37802ad (v7.6.0: action.yml, README.md), pytest-xdist 3.8.0 docs (https://pytest-xdist.readthedocs.io/en/stable/), the REST workflow-runs endpoint (https://docs.github.com/en/rest/actions/workflow-runs#get-a-workflow-run), and our own runs (gh api repos/yellow-pine/noggin/actions/runs/<id>/jobs).

  • Scope: a run restores caches of its own branch or the default branch; a PR run also its base branch. A PR run’s own save goes to refs/pull/<n>/merge and only that PR’s re-runs see it. So only main’s saves seed other PRs and the queue.
  • Lookup: exact key in the run’s branch, then restore-keys prefixes (the most recent match), then the same on the default branch. An exact hit sets cache-hit to true; a restore-keys match restores but leaves it false. A key is written once.
  • ours-convex-: restore-keys: ours-convex- lets a PR that changes a part (or uv.lock) start from main’s latest entry; make ours and mjcf.convex rebuild only what their .key files miss.
  • setup-uv: prune-cache defaults to true, which runs uv cache prune --ci before saving: it keeps only wheels built from source. Every wheel we install is pre-built, so main’s entry held ~1.2 KB and every run downloaded everything. setup-uv has no restore-keys; its key hashes uv.lock and pyproject.toml, and it saves only on success.
  • -n auto: one worker per CPU (4 on a public repo’s ubuntu-latest). --dist loadgroup sends every test with the same xdist_group mark to one worker and spreads the rest: model/test_mjcf.py groups by document, so each worker does not build and sweep every document again. --durations=N reports across workers.
  • Each worker sets PYTEST_XDIST_WORKER (gw0, …). Shared files: mjcf.write truncates and rewrites model/build/mjcf/<doc>.xml while another worker may load it, so model/conftest.py gives each worker its own folder. mjcf.convex decomposes a stale part into a shared folder, so a cold model job decomposes everything first (python -m model.mjcf, its own process pool) and the tests only read.
  • The run’s start: run_started_at of GET /repos/{owner}/{repo}/actions/runs/{run_id} (the latest attempt’s start; needs actions: read). Each job’s last step compares it with now; the run’s wall time is its last job’s.
  • Changed after main’s run of #238 (2b2db76) waited 20 min for runners behind the queue and failed every budget: the clock starts at the earliest started_at of the attempt’s jobs that ran (GET .../runs/{run_id}/attempts/{n}/jobs; queued, waiting, skipped, cancelled and earlier attempts’ jobs left out), and on main an over-budget run warns instead of failing (Cansin).
  • Changed after #265’s “re-run failed jobs” failed its budget at 239 s for 73 s of work (#267): that attempt’s jobs list repeats the jobs it kept with the new run_attempt but their old started_at, so the clock now starts at the first job that ran from the attempt’s run_started_at (GET .../runs/{run_id}/attempts/{n}) on.
  • Changed after 14 Site Checks re-runs in 3.5 h (#438, #442): an over-budget PR run warns like main’s instead of failing; a timing budget is not a correctness gate.
  • Warm: the parts cache hit its exact key (actions/cache cache-hit). A restore-keys match is cold: some parts are rebuilt.
  • -n auto gives 4 workers on ubuntu-latest (no psutil: os.sched_getaffinity). Warm, the model tests take ~194 s on 4 workers with the 4 cores saturated (the durations sum to ~720 s): xdist alone cannot bring the model job under 3 min, so it runs as two shards (model hero, model mujoco) behind one required model check.
  • mjcf.spec starts a spawn process pool for every document it builds, and with --dist worksteal each xdist worker built and swept each document again: every test_mjcf.py test took ~20 s, and the 47 non-hero model tests 165 s.
  • The unpruned uv cache is 448 MB and restores in ~7 s; uv sync then takes 0.2 s. Without a cache setup-uv takes ~1 s and uv sync downloads every wheel in ~3 s. So check.yml keeps no uv cache: it would be slower, and each uv.lock change would put another 448 MB into the repo’s 10 GB cache, evicting the parts and Blender entries.
byYellow PineYellow Pine

noggin v0.2.26-6-g09bf9b7 · 09bf9b7