We shipped Keycloak kit 1.0.0 broken

On 11 October 2026 the Keycloak kit's CI ran all its test suites against the 1.0.0 commit and reported FAILS=0. The release job then pushed the image. Fourteen minutes later, Reputo's dev deploy stopped at provisioning:

bash: /opt/microsaas/keycloak/setup: Permission denied

What was broken, and why the tests passed

The directory /opt/microsaas/keycloak had mode 644. Without the execute bit, Keycloak's user, uid 1000, could not enter it, so neither setup nor require-terms could run. The Dockerfile never created that directory itself: a COPY --chmod=644 of the Slovenian messages file created it as a parent, and the mode of such a parent is up to the BuildKit that runs the build.

Two BuildKits built 1.0.0. The test job ran docker build, with Docker's built-in BuildKit, and got 755. The release job ran buildx with the moby/buildkit image it pulls by default, unpinned and at that time v0.33.1, and got 644. One commit, one Dockerfile, two images. Every suite passed on the image we did not ship, and nothing tested the one we did.

Reputo's dev environment kept serving, because its realm lives in the database. Only Reputo and Leadgener had pinned 1.0.0, and neither runs in production yet.

Test the artifact you ship

The rule we adopted: test the artifact you ship, built by the builder that ships it. The main fix was merged that evening, and 1.0.1 was published the next morning. What changed in CI:

• One BuildKit, pinned by version and digest in release/builder.sh, builds both the image the tests run on and the image that is pushed. The Dockerfile frontend is pinned by digest too.
• The Dockerfile creates every kit directory with mode 755 before any file is copied in, so the modes no longer depend on the builder.
• The structure suite checks, as uid 1000, that every directory can be entered and every file read or run.
• After the push, the release job pulls what the registry serves and tests that. Since 1.0.2 it compares every path under /opt/microsaas, with its mode and owner, in both the amd64 and arm64 images against a fixed list, and creates the version's tag only after both images pass.

For a team building on the toolkit, every Keycloak kit version since 1.0.2 was tested on an image from the builder that pushed it, then checked again as the image you pull, on both platforms. You take its digest from its release notes, and a version that fails the check after its push gets neither notes nor tag: its changelog section is marked defective, and the fix ships as the next patch.