forked from RoushTech/docker
Give each CI run its own compose project so runs stop deleting each other #43
Loading…
Reference in a new issue
No description provided.
Delete branch "ci/unique-compose-project-name"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
./testrunsdocker compose -f test.ymlwith no project name, so compose derives one from the working directory —docker, identically for every branch and every run, on a daemon they all share.Two consequences
Container names collide. From run 299:
And runs delete each other.
test's EXIT trap is:against that shared project. Any run that finishes — or fails early — removes the containers of every other run still in flight. The victim dies mid-build having emitted no error of its own.
Run 291 is precisely that: 4863 lines of healthy build output, then the log simply stops, then a failed job with no error marker anywhere in it. That signature is indistinguishable from the disk-pressure and buildkit-GC contention CLAUDE.md describes, and it has been read as that more than once.
This has never reproduced locally because anyone working from a git worktree already gets a unique project name for free, from the worktree's directory.
The change
COMPOSE_PROJECT_NAME=docker-<run id>, falling back to the pid off CI, with an explicit value from the caller still winning.Isolation has a price, and
reap-buildkit-orphanspays it: a hard-killed run can no longer clean up after itself, and unlike the old shared project, nothing else will ever adopt what it leaves. So the reaper now also collects abandoned test projects.Selection is deliberately conservative, because this daemon is a live Swarm manager:
--filter name=docker-would match half the daemondocker-<digits>shape./testgeneratesTesting
./testunder a unique project name: passes, and leaves nothing behind.docker-12345), local (docker-<pid>), and caller override.now— the most aggressive case possible — across 30 containers in 7 compose projects. It selected the single planted abandoned container and nothing else. The 7 real projects (printshop,someones-edge,surgeon,sw, …) were all spared.Why this is worth landing before anything else
Four sessions are currently pushing to this repo, each triggering a full-matrix build. Runs 297, 298 and 299 all failed inside that window; 300 passed once it thinned out. Until this lands, every run here is exposed to being torn down by an unrelated one, and every resulting failure looks like a bug in whatever change happened to be under test.
Further evidence from the runs that have gone through since this was opened. The shared project name is now confirmed to break runs in two distinct ways, not one.
Run 302 — the name collision:
Run 301 — the shared network deleted mid-run, which I had not seen before:
docker_defaultis the default network of the shareddockerproject. A sibling run'sdown --remove-orphans --volumesremoved it while run 301 was still attaching containers to it. Same root cause as the conflict, opposite symptom, and this one produces an error that points nowhere near the real problem.So the tally on this daemon is now:
container name ... already in usenetwork docker_default not foundcontainer name ... already in useWorth noting how the recent record reads without this context: of runs 295–302, the only two that passed (295 and 300) were both
No image inputs changed; nothing to test— they built nothing. Every run in that window that actually built something failed, and each failure looked like a defect in whatever change happened to be under test.One useful property while this rolls out: the fix is unilaterally protective. A run on this branch is already immune to a sibling on
maintearing it down, because the sibling'sdowntargets projectdockerand this one lives indocker-<run id>. It does not need every branch to adopt it before it starts helping.