ci: reap the BuildKit builders CI has been leaking #21

Merged

geusebio merged 1 commit from ci/reap-orphaned-buildkit-builders into main

2026-08-04 11:52:00 +00:00

geusebio commented

2026-08-04 10:32:10 +00:00

Owner

Main and every open PR have been failing on the runner's disk rather than on anything in the diff.

runs 254, 255:  no space left on device
runs 256, 257:  Internal: unknown blob sha256:... in history

The second is the first in disguise: once free space drops far enough, buildkit's GC evicts blobs while an export still references them, so the build dies mid importing to docker instead of with a clean ENOSPC.

What leaks

docker/setup-buildx-action creates a throwaway docker-container builder per job and removes it in its post step. On a loaded daemon that removal times out — from run 254:

failed to remove builder-6efb...: failed to remove node builder-6efb...0:
Delete "http://forgejo-docker-socket:2375/v1.51/containers/buildx_buildkit_builder-6efb...0?force=1&v=1": context deadline exceeded

The buildkit container and its state volume are orphaned permanently, and each holds a full build cache — the builder advertises maxUsedSpace: 90.34GiB. Nothing reaps them, so the floor rises with every timed-out cleanup.

The fix

./reap-buildkit-orphans, run at the top of both jobs.

It is deliberately narrow. The daemon behind forgejo-docker-socket is a live Swarm manager — docker info reports 87 containers, 178 images, Is Manager: true — so it touches only objects named buildx_buildkit_*. No system prune, no image prune, no blanket volume prune.

Builders that have exited are removed immediately: buildx keeps a builder's container running for as long as the builder exists, so a stopped one is unambiguously dead.
Builders still running are removed only after 6 hours — past the test job's 60-minute cap and the build job's 5×60 bake retry envelope, so a peer job's live builder is never in scope.
Volumes are filtered on dangling=true, so one still attached to a live builder is never listed.
docker system df prints unconditionally, so the next failure says whether the leak came back or something else filled the disk.

Also

CLAUDE.md gains the daemon's shared, non-disposable nature, plus two monitoring corrections found while diagnosing this: /actions/tasks hangs and times out (use /actions/runs, filtered on commit_sha), and the API wants a run's internal id where the web UI shows index_in_repo — POST /actions/runs/{id}/cancel 404s if given the run number.

Main and every open PR have been failing on the runner's disk rather than on anything in the diff. ``` runs 254, 255: no space left on device runs 256, 257: Internal: unknown blob sha256:... in history ``` The second is the first in disguise: once free space drops far enough, buildkit's GC evicts blobs while an export still references them, so the build dies mid `importing to docker` instead of with a clean ENOSPC. ## What leaks `docker/setup-buildx-action` creates a throwaway `docker-container` builder per job and removes it in its post step. On a loaded daemon that removal times out — from run 254: ``` failed to remove builder-6efb...: failed to remove node builder-6efb...0: Delete "http://forgejo-docker-socket:2375/v1.51/containers/buildx_buildkit_builder-6efb...0?force=1&v=1": context deadline exceeded ``` The buildkit container and its state volume are orphaned permanently, and each holds a full build cache — the builder advertises `maxUsedSpace: 90.34GiB`. Nothing reaps them, so the floor rises with every timed-out cleanup. ## The fix `./reap-buildkit-orphans`, run at the top of both jobs. It is deliberately narrow. The daemon behind `forgejo-docker-socket` is a **live Swarm manager** — `docker info` reports 87 containers, 178 images, `Is Manager: true` — so it touches only objects named `buildx_buildkit_*`. No `system prune`, no `image prune`, no blanket `volume prune`. - Builders that have **exited** are removed immediately: buildx keeps a builder's container running for as long as the builder exists, so a stopped one is unambiguously dead. - Builders still **running** are removed only after 6 hours — past the test job's 60-minute cap and the build job's 5×60 bake retry envelope, so a peer job's live builder is never in scope. - Volumes are filtered on `dangling=true`, so one still attached to a live builder is never listed. - `docker system df` prints unconditionally, so the next failure says whether the leak came back or something else filled the disk. ## Also CLAUDE.md gains the daemon's shared, non-disposable nature, plus two monitoring corrections found while diagnosing this: `/actions/tasks` hangs and times out (use `/actions/runs`, filtered on `commit_sha`), and the API wants a run's internal `id` where the web UI shows `index_in_repo` — `POST /actions/runs/{id}/cancel` 404s if given the run number.

geusebio added 1 commit

2026-08-04 10:32:10 +00:00

ci: reap the BuildKit builders CI has been leaking

Build / Build (push) Has been cancelled

Details

Build / Run container tests (push) Has been cancelled

Details

6ff8ae3c4b

Both failure modes on main come from the runner's disk, not from any
change under test:

  runs 254, 255:  no space left on device
  runs 256, 257:  Internal: unknown blob sha256:... in history

The second is the first wearing a disguise. Once free space drops far
enough, buildkit's GC starts evicting blobs while an export still holds
a reference to them, and the build dies mid "importing to docker" rather
than with a clean ENOSPC.

What fills the disk is docker/setup-buildx-action. It creates a
throwaway `docker-container` builder per job and removes it in its post
step, but on a loaded daemon that removal times out -- from run 254:

  failed to remove builder-6efb...: failed to remove node
  builder-6efb...0: Delete "http://forgejo-docker-socket:2375/v1.51/
  containers/buildx_buildkit_builder-6efb...0?force=1&v=1":
  context deadline exceeded

The buildkit container and its state volume are then orphaned for good,
and each one holds a full build cache -- the builder advertises
maxUsedSpace 90.34GiB. Nothing reaps them, so the floor rises with every
timed-out cleanup until builds stop fitting.

./reap-buildkit-orphans runs at the top of both jobs and removes them.
It is deliberately narrow: the daemon behind forgejo-docker-socket is a
live Swarm manager (87 containers, 178 images, Is Manager: true), so it
touches only objects named buildx_buildkit_* -- no system prune, no
image prune, no blanket volume prune. Builders that have exited go
immediately; running ones only after six hours, which is past the test
job's 60-minute cap and the build job's 5x60 bake retry envelope, so a
peer job's live builder is never in scope. Volumes are filtered on
dangling=true, so one still attached to a live builder is never listed.

It also prints `docker system df` unconditionally, so the next failure
says whether the leak came back or something else filled the disk.

CLAUDE.md picks up the daemon's shared, non-disposable nature, and two
monitoring corrections found while diagnosing this: the actions/tasks
endpoint hangs (use actions/runs, filtered on commit_sha), and the API
wants a run's internal `id` where the web UI shows `index_in_repo`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

geusebio force-pushed ci/reap-orphaned-buildkit-builders from 6ff8ae3c4b

Build / Build (push) Has been cancelled

Details

Build / Run container tests (push) Has been cancelled

Details

to 6be95cd0a7

Build / Run container tests (push) Failing after 41m50s

Details

Build / Build (push) Has been skipped

Details

2026-08-04 10:36:16 +00:00

Compare