ko

were shipping Go programs, `ko` would likely be a good baseline.

vastly more useful traces than in Go, and unfortunately too many programmers go mininum-effort path. [0] https://github.com/ko-build/ko/blob/cfc13deeb6417d7e1582f031...

ko (https://github.com/google/ko), which can achieve reproducible builds for go projects without much fuss. It also leans heavily on go's excellent support for cross-compilation to produce multi-platform images, trivially. > There are two cases where the container-diff tool will report that the registry and local images that you are comparing are the same (in terms of Docker history lines), but will be misleading because the images are actually different. While container-diff is great, it can obscure what's really going on a bit. If you're interested in uncovering exactly why the digest of the image you built is different from what was published, please forgive another shameless plug for crane (https://github.com/google/go-containerregistry/blob/main/cmd...), a tool I wrote to expose most of the functionality of go-containerregistry (https://github.com/google/go-containerregistry), which is the library both container-diff and ko use under the hood. Forgive the sparse documentation, but it should be relatively straightforward for anyone familiar with the registry API and data structures, as the commands map pretty directly to registry functionality. Using crane, you can easily inspect the image in the registry directly to compare the manifests and blobs that make up an image. For example, one reason that the digests might never match is that these images are somewhat strangely wrapped as singleton manifest lists: https://gist.github.com/jonjohnsonjr/ffba104ca504b5bb4a1f227... It makes some sense to me that they might want to do this to prevent folks from pulling this on windows, but usually you would only encounter manifest lists for multi-platform images. Even if these builds were reproducible, you would have to compare the digest of what you built with sha256:9a210bb9cbbdba5ae2199b659551959cd01e0299419f4118d111f8443971491a -- not the sha256:fb1a43b50c7047e5f28e309268a8f5425abc9cb852124f6828dcb0e4f859a4a1 that docker outputs, as shown in the article. The tag used for this example (mcr.microsoft.com/dotnet/sdk:5.0-alpine) has since been updated. Comparing this with the original using container-diff just tells us that the size changed: https://gist.github.com/jonjohnsonjr/90c2def551833c8cacf3264... But looking at the actual manifests, config blobs, and layers using crane is often faster and more interesting: https://gist.github.com/jonjohnsonjr/283eab27d996b2f4cc04553... My intention with crane is to be easily composable so that you can use familiar tools like tar, sort, diff, jq, etc. (To be fair to container-diff, you can use the -t flag to show similar things.) I realize this is not really the point of the article, but it's a huge pet peeve of mine that everyone has just given up on understanding what's going on with their images because the tooling UX makes everything so opaque. If the digest of something doesn't match, you should know why! It's as if `git push --force` was on by default and everyone has just accepted that reality. Now to read the rest of the article :)