ci: only rebuild the images a change actually affects #20

Merged

geusebio merged 1 commit from ci/scope-builds-to-changed-images into ci/drop-unsupported-permissions-blocks

2026-08-04 13:00:36 +00:00

geusebio commented

2026-08-04 09:56:51 +00:00

Owner

Stacked on #18 (its build.yml changes are the base for these). Retarget to main once that merges.

Why

Measured from the job logs rather than guessed:

Build job ~30 min, test job 6-26 min, run sequentially.
The cost is package downloads, not CPU and not QEMU. dotnet-10 takes 1191s on native amd64 vs 1288s on arm64 under emulation -- an 8% delta. Inside that step, 1070 MiB installs with multi-hundred-second stalls between individual packages.
In both jobs a single apk add is the critical path (test run 234: INSTALL_JAVA at 1060s of a 26 min run; the same step took 167s in run 232 -- same inputs, 6x spread, pure mirror variance).

Most of that work is for images the commit never touched.

What

changed-targets works out what a change affects. The graph is parsed from the Dockerfiles themselves (INCLUDE ./x, FROM ./x#stage, COPY ./fs/y); target names come from bake --print and compose config. Adding an image needs no edit to it and the mapping cannot drift.
Tuesday 04:00 UTC schedule still builds everything, so rarely-touched images keep getting rebuilt against current upstream packages.
./test grows TEST_SERVICES. It generates a test-subset aggregator rather than an override file, because compose merges depends_on by union and so can only ever add services to test, never narrow it.
Concurrency group fixed: github.head_ref is only set for pull_request, so on a push it fell back to the unique run_id and cancel-in-progress never fired. The weekly run gets its own lane so a Tuesday-morning push cannot cancel the only build covering every image.

Measured

Change	Builds	Tests
`fs/dotnet/...`	dotnet-9, dotnet-10	nothing
`PHP.fragment.Dockerfile`	php + magento	32 services, 0 dotnet
`Java.fragment.Dockerfile`	java + tomcat + war-machine	10 services
`fs/alpine/...`	everything	everything
`README.md`	nothing	nothing

A 2-service subset ran in 31s vs 6-26 min for the full suite.

Fallbacks bias toward overbuilding: no BASE_SHA, unreachable commit, empty diff, or a change to docker-bake.hcl/test*.yml/the scripts/.github/ all force a full build. In the build job "nothing affected" is an explicit NONE sentinel, so a dropped variable fails loudly instead of silently building nothing.

Verification

Passing: closure logic against 8 synthetic diffs and 7 real scoped commits in a throwaway worktree; subset runs; both no-op paths; act -n; YAML parse.

Not verified locally: a green full-suite run. It fails at 27s with frontend grpc server closed unexpectedly on the dockerfile-x syntax frontend. The unmodified script from HEAD fails identically at 26s, so this is a pre-existing local Docker Desktop issue rather than this change -- but the full-suite path rests on inspection plus that comparison, and on CI here.

🤖 Generated with Claude Code

Stacked on #18 (its build.yml changes are the base for these). Retarget to `main` once that merges. ## Why Measured from the job logs rather than guessed: - Build job ~30 min, test job 6-26 min, run sequentially. - The cost is **package downloads, not CPU and not QEMU**. `dotnet-10` takes 1191s on *native* amd64 vs 1288s on arm64 under emulation -- an 8% delta. Inside that step, 1070 MiB installs with multi-hundred-second stalls between individual packages. - In both jobs a single `apk add` is the critical path (test run 234: `INSTALL_JAVA` at 1060s of a 26 min run; the same step took 167s in run 232 -- same inputs, 6x spread, pure mirror variance). Most of that work is for images the commit never touched. ## What - `changed-targets` works out what a change affects. The graph is parsed from the Dockerfiles themselves (`INCLUDE ./x`, `FROM ./x#stage`, `COPY ./fs/y`); target names come from `bake --print` and `compose config`. Adding an image needs no edit to it and the mapping cannot drift. - Tuesday 04:00 UTC schedule still builds everything, so rarely-touched images keep getting rebuilt against current upstream packages. - `./test` grows `TEST_SERVICES`. It generates a `test-subset` aggregator rather than an override file, because compose merges `depends_on` by union and so can only ever *add* services to `test`, never narrow it. - Concurrency group fixed: `github.head_ref` is only set for `pull_request`, so on a push it fell back to the unique `run_id` and `cancel-in-progress` never fired. The weekly run gets its own lane so a Tuesday-morning push cannot cancel the only build covering every image. ## Measured | Change | Builds | Tests | |---|---|---| | `fs/dotnet/...` | dotnet-9, dotnet-10 | *nothing* | | `PHP.fragment.Dockerfile` | php + magento | 32 services, **0 dotnet** | | `Java.fragment.Dockerfile` | java + tomcat + war-machine | 10 services | | `fs/alpine/...` | everything | everything | | `README.md` | nothing | nothing | A 2-service subset ran in 31s vs 6-26 min for the full suite. Fallbacks bias toward overbuilding: no `BASE_SHA`, unreachable commit, empty diff, or a change to `docker-bake.hcl`/`test*.yml`/the scripts/`.github/` all force a full build. In the build job "nothing affected" is an explicit `NONE` sentinel, so a dropped variable fails loudly instead of silently building nothing. ## Verification Passing: closure logic against 8 synthetic diffs and 7 real scoped commits in a throwaway worktree; subset runs; both no-op paths; `act -n`; YAML parse. **Not verified locally:** a green full-suite run. It fails at 27s with `frontend grpc server closed unexpectedly` on the `dockerfile-x` syntax frontend. The unmodified script from HEAD fails identically at 26s, so this is a pre-existing local Docker Desktop issue rather than this change -- but the full-suite path rests on inspection plus that comparison, and on CI here. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

geusebio added 1 commit

2026-08-04 09:56:51 +00:00

ci: only rebuild the images a change actually affects

Build / Run container tests (push) Failing after 20m47s

Details

Build / Build (push) Has been skipped

Details

0c538a2f7b

The build was spending ~30 minutes rebuilding every image on every push,
and the test job another 6-26. Measuring the job logs showed the cost is
package downloads, not CPU: dotnet-10 takes 1191s on native amd64 versus
1288s on arm64 under QEMU, and the gaps sit between individual apk
installs. Most of that work is for images the commit never touched.

Scope each push to the images whose inputs changed, and add a Tuesday
04:00 run that still builds everything, so images nothing depends on --
dotnet above all, ~20 minutes on its own -- keep getting rebuilt against
current upstream packages.

changed-targets derives the graph from the Dockerfiles themselves
(INCLUDE ./x, FROM ./x#stage, COPY ./fs/y) and reads target names back
out of `bake --print` and `compose config`, so adding an image needs no
change to it and the mapping cannot drift. Files that cannot be scoped
that way -- the bake file, the test compose files, the scripts,
.github/ -- force a full build, as does an unknown or unreachable base
commit. Getting this wrong should cost time, never correctness.

./test grows TEST_SERVICES. It builds a generated test-subset aggregator
rather than an override file because compose merges depends_on by union,
so an override can only ever add services to `test`, never narrow it.

Also fix the concurrency group. `github.head_ref` is only set for
pull_request events, so on a push it fell back to the always-unique
run_id and cancel-in-progress never cancelled anything. The weekly run
gets its own lane so a Tuesday-morning push cannot cancel the only build
that covers every image.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

geusebio force-pushed ci/scope-builds-to-changed-images from 0c538a2f7b

Build / Run container tests (push) Failing after 20m47s

Details

Build / Build (push) Has been skipped

Details

to 10f68383f5

Build / Run container tests (push) Successful in 18m27s

Details

Build / Build (push) Successful in 21m2s

Details

2026-08-04 12:59:00 +00:00

Compare