go-containerregistry

change, which led to a major SemVer update—but it was eventually necessary. We needed to completely rewrite how we manage customer organizations. In Distr, we differentiate between vendors and customers. A vendor is typically the author of a software / AI application that wants to distribute it to customers. Previously, we had taken a shortcut where every customer was just a single user who owned a deployment. We’ve now introduced customer organizations. Vendors onboard customer organizations onto the platform, and customers own their internal user management, including RBAC. This change obviously broke our API, and although the migration for our cloud customers was smooth, custom solutions built on top of our APIs needed updates. Other notable features we’ve implemented since our first launch: - An OCI container registry built on an adapted version of https://github.com/google/go-containerregistry/, directly embedded into our codebase and served via a separate port from a single Docker image. This allows vendors to distribute Docker images and other OCI artifacts if customers want to self-manage deployments. - License Management to restrict which customers can access which applications or artifact versions. Although “license management” is a broadly used term, the main purpose here is to codify contractual agreements between vendors and customers. In its simplest form, this is time-based access to specific software versions, which vendors can now manage with Distr. - Container logs and metrics you can actually see without SSH access. Internally, we debated whether to use a time-series database or store all logs in Postgres. Although we had to tinker quite a bit with Postgres indexes, it now runs stably. - Secret Management, so database passwords don’t show up in configuration steps or logs. Distr is now used by 200+ vendors, including Fortune 500 companies, across on-prem, GovCloud, AWS, and GCP, spanning health tech, fintech, security, and AI companies. We’ve also started working on our first air-gapped environment. For Distr 3.0, we’re working on native Terraform / OpenTofu and Zarf support to provision and update infrastructure in customers’ cloud accounts and physical environments—empowering vendors to offer BYOC and air-gapped use cases, all from a single platform. Distr is fully open source and self-hostable: https://github.com/distr-sh/distr Docs: https://distr.sh/docs We’re YC S24. Happy to answer questions about on-prem deployments and would love to hear about your experience with complex customer deployments.

change, which led to a major SemVer update—but it was eventually necessary. We needed to completely rewrite how we manage customer organizations. In Distr, we differentiate between vendors and customers. A vendor is typically the author of a software / AI application that wants to distribute it to customers. Previously, we had taken a shortcut where every customer was just a single user who owned a deployment. We’ve now introduced customer organizations. Vendors onboard customer organizations onto the platform, and customers own their internal user management, including RBAC. This change obviously broke our API, and although the migration for our cloud customers was smooth, custom solutions built on top of our APIs needed updates. Other notable features we’ve implemented since our first launch: - An OCI container registry built on an adapted version of https://github.com/google/go-containerregistry/, directly embedded into our codebase and served via a separate port from a single Docker image. This allows vendors to distribute Docker images and other OCI artifacts if customers want to self-manage deployments. - License Management to restrict which customers can access which applications or artifact versions. Although “license management” is a broadly used term, the main purpose here is to codify contractual agreements between vendors and customers. In its simplest form, this is time-based access to specific software versions, which vendors can now manage with Distr. - Container logs and metrics you can actually see without SSH access. Internally, we debated whether to use a time-series database or store all logs in Postgres. Although we had to tinker quite a bit with Postgres indexes, it now runs stably. - Secret Management, so database passwords don’t show up in configuration steps or logs. Distr is now used by 200+ vendors, including Fortune 500 companies, across on-prem, GovCloud, AWS, and GCP, spanning health tech, fintech, security, and AI companies. We’ve also started working on our first air-gapped environment. For Distr 3.0, we’re working on native Terraform / OpenTofu and Zarf support to provision and update infrastructure in customers’ cloud accounts and physical environments—empowering vendors to offer BYOC and air-gapped use cases, all from a single platform. Distr is fully open source and self-hostable: https://github.com/distr-sh/distr Docs: https://distr.sh/docs We’re YC S24. Happy to answer questions about on-prem deployments and would love to hear about your experience with complex customer deployments.

e image first is to support "RUN" build steps that execute a command inside of a container and store the filesystem changes in a new layer. If that functionality is ever added to rules_img, I expect it to have the same performance as other build tools. [0]: https://github.com/google/go-containerregistry/blob/main/cmd... [1]: https://regclient.org/cli/regctl/image/mod/

they're doing, since they're not able to do "whatever the user wants". 1. https://github.com/google/go-containerregistry is a Go module to interact with images in the registry and in tarballs and layouts, in the local docker daemon. You can append layers, squash layers, modify metadata, etc. This library is used by all kinds of stuff, including buildpacks, Bazel's rules_docker, and all of the below, to build images without Docker. 2. crane is a CLI that uses the above (in the same repo) to make many of the same modifications from the commandline. `crane append` for instance adds a layer containing some contents to an image, entirely in the registry, without even pulling the base image. 3. ko (https://ko.build) is a tool to build Go applications into images without Dockerfiles or Docker at all. It runs `go build`, appends that binary on top of a base image, and pushes it directly to the registry. It generates an SBOM declaring what Go modules went into the app it put into the image, since that's all it can do. 4. apko (https://apko.dev) is a tool to assemble an image from pre-built apks, without Docker. It's capable of producing "distroless" images easily with config in YAML. It generates an SBOM declaring exactly what apks it put in the image, since that's all it can do. Bazel's rules_docker is another contender in the space, and GCP's distroless images use it to place Debian .debs into an image. Apko is its spiritual successor, and uses YAML instead of Bazel's own config language, which makes it a lot easier to adopt and use (IMO), with all of the same benefits. I'm excited to see more folks realizing that Dockerfiles aren't always necessary, and can sometimes make your life harder. I'm extra excited to see more tools and tutorials digging into the details of how container images work, and preaching the gospel that they can be built and modified using existing tooling and relatively simple libraries. Excellent article!

ko (https://github.com/google/ko), which can achieve reproducible builds for go projects without much fuss. It also leans heavily on go's excellent support for cross-compilation to produce multi-platform images, trivially. > There are two cases where the container-diff tool will report that the registry and local images that you are comparing are the same (in terms of Docker history lines), but will be misleading because the images are actually different. While container-diff is great, it can obscure what's really going on a bit. If you're interested in uncovering exactly why the digest of the image you built is different from what was published, please forgive another shameless plug for crane (https://github.com/google/go-containerregistry/blob/main/cmd...), a tool I wrote to expose most of the functionality of go-containerregistry (https://github.com/google/go-containerregistry), which is the library both container-diff and ko use under the hood. Forgive the sparse documentation, but it should be relatively straightforward for anyone familiar with the registry API and data structures, as the commands map pretty directly to registry functionality. Using crane, you can easily inspect the image in the registry directly to compare the manifests and blobs that make up an image. For example, one reason that the digests might never match is that these images are somewhat strangely wrapped as singleton manifest lists: https://gist.github.com/jonjohnsonjr/ffba104ca504b5bb4a1f227... It makes some sense to me that they might want to do this to prevent folks from pulling this on windows, but usually you would only encounter manifest lists for multi-platform images. Even if these builds were reproducible, you would have to compare the digest of what you built with sha256:9a210bb9cbbdba5ae2199b659551959cd01e0299419f4118d111f8443971491a -- not the sha256:fb1a43b50c7047e5f28e309268a8f5425abc9cb852124f6828dcb0e4f859a4a1 that docker outputs, as shown in the article. The tag used for this example (mcr.microsoft.com/dotnet/sdk:5.0-alpine) has since been updated. Comparing this with the original using container-diff just tells us that the size changed: https://gist.github.com/jonjohnsonjr/90c2def551833c8cacf3264... But looking at the actual manifests, config blobs, and layers using crane is often faster and more interesting: https://gist.github.com/jonjohnsonjr/283eab27d996b2f4cc04553... My intention with crane is to be easily composable so that you can use familiar tools like tar, sort, diff, jq, etc. (To be fair to container-diff, you can use the -t flag to show similar things.) I realize this is not really the point of the article, but it's a huge pet peeve of mine that everyone has just given up on understanding what's going on with their images because the tooling UX makes everything so opaque. If the digest of something doesn't match, you should know why! It's as if `git push --force` was on by default and everyone has just accepted that reality. Now to read the rest of the article :)