forked from RoushTech/docker
ci: stop main pushes cancelling the publish in flight #33
Loading…
Reference in a new issue
No description provided.
Delete branch "ci/dont-cancel-main-publishes"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Investigating #26 (the
FRANKENPHP_MERCUREbuild never reaching Docker Hub) turned up two causes. The first — leaked BuildKit builders exhausting the runner disk — was #21, now merged. This is the second, and it is why main still cannot publish.What happens
Forgejo populates
github.head_refon push events, unlike GitHub where it is only set forpull_request. The original grouptherefore never reached its always-unique
run_idfallback — it resolved to the branch name, so a push cancelled whatever was already running on that same ref. #20'sgithub.refkey is equivalent for branches and preserves that.On a feature branch that is exactly right. On main it means each merge kills the previous merge's build:
Same signature on branches (run 268 by the next push to
fix/frankenphp-zts-stack-size, run 262 onci/reap-orphaned-buildkit-builders), which is what confirms the group is per-ref rather than per-run.Run 265 is the damaging one. It had passed the test gate, reached the bake, authenticated, and was uploading when it died:
A publish cancelled mid-push leaves some targets updated and some not — which is how the registry ended up holding an incomplete set of build-stamped tags rather than a clean set. With merges landing roughly every 15 minutes and a full pipeline taking ~50, main could not finish a publish at all.
The change
Main keys on the commit sha, so every main push gets its own group and builds queue instead of cancelling each other. Branches keep keying on the ref and still cancel superseded runs. #20's weekly schedule lane is preserved as-is.
cancel-in-progressstays a literaltruerather than becoming an expression — this runner is fussy enough about expression contexts that it seemed worth keeping the conditional entirely inside the group string, a form #20 already proved works here.Validated with
act -n -W .github/workflows/build.yml; both jobs still parse and resolve.Note on scoping
#20's
./changed-targetsmeans a push only rebuilds what its own diff touches. Combined with cancellation that has a sharp edge: if a run is cancelled, the images its diff touched are never built, and the next push scopes against its owngithub.event.before— so those images can be skipped indefinitely rather than picked up later. This PR removes the cancellation half on main; the scoping interaction is worth a separate look, and I have not touched it here.Closes #26
#26 is only genuinely resolved once a fresh image on Docker Hub actually contains the Mercure support, so I will verify the published digest and
/etc/services.d/frankenphp/runcontents after this lands rather than treating the merge as the finish line.Forgejo populates `github.head_ref` on push events, unlike GitHub where it is only set for pull_request. The original `${{ github.head_ref || github.run_id }}` group therefore never reached its always-unique fallback -- it resolved to the branch name, so a push cancelled whatever was already running on that ref. Keying on `github.ref` (#20) is equivalent for branches and keeps that behaviour. Cancelling superseded runs is right on a feature branch and wrong on main, where it means each merge kills the previous merge's build: run 265 main80b9ddbcancelled 7s afterc18fa74landed, partway through `pushing layers` to Docker Hub run 270 mainc18fa74cancelled whena5e7b76landed 265 is the damaging case: a publish cancelled mid-push leaves some targets updated and some not, which is how Docker Hub ended up holding an incomplete set of build-stamped tags. With merges landing every ~15 minutes and a full pipeline taking ~50, main could not finish a publish at all -- which is why the FRANKENPHP_MERCURE build never shipped. Key main on the commit sha so every main push gets its own group and builds queue instead of cancelling each other. Branches keep keying on the ref and still cancel superseded runs. The weekly schedule lane from #20 is preserved. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>Following up on this after watching it land — the fix is not proven, and there is a specific way it could be inert. Flagging rather than leaving it looking settled.
Three merges hit main within 30 seconds, all three carrying this change:
Both cancelled runs had started (284 was mid-checkout retry, 285 had logged
Login Succeededand reaped orphans) and both end incontext canceled. Under this change their groups should have beenBuild-7732730…-ondemandandBuild-8996e2c…-ondemand— distinct from each other and from 286. They should not have been cancellable by a sibling.Two explanations fit, and I cannot separate them from the data I have:
github.shais empty in the concurrency context. The expression would then collapse toBuild--ondemand, shared across every main push — which reproduces exactly the observed pattern of "newest survives, older ones die", and would mean this change is inert. Given the runner has already surprised us once onhead_ref, assuming GitHub's context semantics hold is precisely the mistake this PR was correcting.concurrencyblock. Then no group expression fixes it and a different approach is needed.Either way, run 286 succeeding is weak evidence for this PR: it was the last push of the burst, and nothing merged for ~30 minutes afterwards. It may simply have been left alone.
How to tell
Next time two merges land on main close together, check whether the earlier run survives. If it is cancelled, this is inert and the next thing to try is
github.run_idin place ofgithub.sha— genuinely unique per run, and its availability here is untested because the originalhead_ref || run_idnever reached the fallback.Happy to take that on. I would rather leave this open as a known-unknown than have it read as done.