buildkit

BuildKit BuildKit is a toolkit for converting source code to build artifacts in an efficient, expressive and repeatable manner. Key features: Automatic garbage collection Extendable frontend formats Concurrent dependency resolution Efficient instruction caching Build cache import/export Nested build job invocations Distributable workers Multiple output formats Pluggable architecture Execution without root privileges Read the proposal from moby/moby#32925 Introductory blog post https://blog.mobyproject.org/introducing-buildkit-17e056cc5317 Join #buildkit channel on Docker Community Slack NoteIf you are visiting this repo for the usage of BuildKit-only Dockerfile features like RUN --mount=type=(bind|cache|tmpfs|secret|ssh), please refer to the Dockerfile reference. Notedocker build uses Buildx and BuildKit by default since Docker Engine 23.0. You don't need to read this document unless you want to use the full-featured standalone version of BuildKit. Used by Quick start Linux Setup Windows Setup macOS Setup Build from source Exploring LLB Frontends Exploring Dockerfiles Building a Dockerfile with buildctl Building a Dockerfile using external frontend Output Image/Registry Local directory Docker tarball OCI tarball containerd image store Cache Garbage collection Export cache Inline (push image and cache together) Registry (push image and cache separately) Local directory GitHub Actions cache (experimental) S3 cache (experimental) Azure Blob Storage cache (experimental) Consistent hashing Metadata Systemd socket activation Expose BuildKit as a TCP service Load balancing Containerizing BuildKit Podman Nerdctl Kubernetes Daemonless OpenTelemetry support Running BuildKit without root privileges Building multi-platform images Configuring buildctl Color Output Controls Number of log lines (for active steps in tty mode) Contributing Used by BuildKit is used by the following projects: Moby & Docker (DOCKER_BUILDKIT=1 docker build) img OpenFaaS Cloud container build interface Tekton Pipelines (formerly Knative Build Templates) the Sanic build tool vab Rio kim PouchContainer Docker buildx Okteto Cloud Earthly earthfiles Gitpod Dagger envd Depot Namespace Unikraft DevZero dacc Quick start ℹ️ For Kubernetes deployments, see examples/kubernetes. BuildKit is composed of the buildkitd daemon and the buildctl client. While the buildctl client is available for Linux, macOS, and Windows, the buildkitd daemon is only available for Linux and *Windows currently. The latest binaries of BuildKit are available here for Linux, macOS, and Windows. Linux Setup The buildkitd daemon requires the following components to be installed: runc or crun containerd (if you want to use containerd worker) Starting the buildkitd daemon: You need to run buildkitd as the root user on the host. $ sudo buildkitd To run buildkitd as a non-root user, see docs/rootless.md. The buildkitd daemon supports two worker backends: OCI (runc) and containerd. By default, the OCI (runc) worker is used. You can set --oci-worker=false --containerd-worker=true to use the containerd worker. We are open to adding more backends. To start the buildkitd daemon using systemd socket activation, you can install the buildkit systemd unit files. See Systemd socket activation The buildkitd daemon listens gRPC API on /run/buildkit/buildkitd.sock by default, but you can also use TCP sockets. See Expose BuildKit as a TCP service. Windows Setup See instructions and notes at docs/windows.md. macOS Setup Homebrew formula (unofficial) is available for macOS. $ brew install buildkit The Homebrew formula does not contain the daemon (buildkitd). For example, Lima can be used for launching the daemon inside a Linux VM. brew install lima limactl start template://buildkit export BUILDKIT_HOST="unix://$HOME/.lima/buildkit/sock/buildkitd.sock" Build from source To build BuildKit from source, see .github/CONTRIBUTING.md. For a buildctl reference, see this document. Exploring LLB BuildKit builds are based on a binary intermediate format called LLB that is used for defining the dependency graph for processes running part of your build. tl;dr: LLB is to Dockerfile what LLVM IR is to C. Marshaled as Protobuf messages Concurrently executable Efficiently cacheable Vendor-neutral (i.e. non-Dockerfile languages can be easily implemented) See solver/pb/ops.proto for the format definition, and see ./examples/README.md for example LLB applications. Frontends BuildKit supports different build languages through frontends that translate build definitions into LLB. The following frontends and tools provide alternatives for defining builds: Project Description Dockerfile Build container images using Dockerfile instructions. Buildpacks Build applications using buildpacks. Mockerfile Example YAML frontend for Ubuntu packages and external dependencies. Gockerfile Build simple Go servers from YAML definitions. bldr (Pkgfile) Build and package software distributions using YAML definitions. HLB Define builds as containerized units of work in a high-level build language. Earthfile (Earthly) Define repeatable builds with Dockerfile-like syntax. No longer actively maintained. Cargo Wharf (Rust) Build Rust container images from Cargo build plans with per-dependency caching. Nix Build container images from Nix derivations and flakes. mopy (Python) Package Python applications into container images using YAML definitions. envd (Starlark) Define container-based development environments for AI/ML. Blubber Build application container images from declarative YAML. Bass Script builds and project infrastructure with a Lisp dialect. kraft.yaml (Unikraft) Build Unikraft unikernel images from Kraftfiles. Massé Define composable container image build graphs in CUE. DALEC Build system packages and container images from declarative specifications. Dexfile Extend Dockerfile syntax with conditions, loops, imports, and functions. Doc