LIlihaoyi·Dec 9, 2024
Bazel does a lot of non-obvious things that other build tools don't, that become important in larger codebases (100-10,000 active developers):
- Automatic caching of everything.
In most build tools, caching is opt in, so the core build/compile steps usually end up getting cached but everything else is not and ends up being wastefully recomputed all the time. In Bazel, caching is the default so everything is cached. Even tests are cached so if you run a test twice on the same code and inputs (transitively), the second time it is skipped
- Dependency based test selection: You can use bazel query to determine the possible targets and tests affected by a code change, allowing you to trivially set up CI to only run tests downstream of a PR diff and skip unrelated ones. Any large codebase that doesn't use Bazel ends up re-inventing this manuall
Read the rest
y, e.g. consider this code in apache/spark that re-implements this in a Python script that wraps `mvn` or `sbt` (which are build tools that do not provide this functionality) https://github.com/apache/spark/blob/290b4b31bae2e02b648d2c5...
- Automatic sandboxing of your build steps in CGroup/NS containers, to ensure your build steps do not make use of un-declared files. In most build tools, this kind of mistake results in confusing nondeterministic parallelism and cache invalidation problems down the road, where e.g. your build step may rely on a file on disk but not realize it needs to re-compute when the file changes. In Bazel, these mis-configurations result in a deterministic error up front
- At my last job we extended these cgroups to limit CPU/Memory usage as well, which eliminates the noisy neighbour problem and ensures a build step or test gets the same compute footprint whether run alone during development or 96x parallel on a CI worker (https://github.com/bazelbuild/bazel/pull/21322). Otherwise it's common for tests to pass when run during manual development then timeout or OOM when run in CI under resource pressure due to other tests hogging the CPU or RAM
- Built in support for seamless shared caches: e.g. I compile something on my laptop, you download it to your laptop for usage. This also applies to tests: e.g. if TestFoo was run in CI on master, if i pull master and run all tests without changing the Foo code, TestFoo is skipped and uses the CI result
- Built in support for shared compute clusters: e.g. I compile stuff and it automatically happens in the cloud on 96 core machines, or i run a lot of tests (e.g. after a big refactor) on my laptop and it automatically gets farmed out to run 1024x parallel on the cluster (which despite running faster shouldn't cost any more than running 1x parallel, since it costs 1024x as much per-second but should finish in 1024x fewer seconds!)
- Support for deep integration between multiple languages: e.g. building a Go binary and Rust library which are both used in a Python executable, which is then tested using a Bash script and deployed as part of a Java backend server. Most build tools support one language well and others not at all, Bazel supports them all (not perfect but adequately)
If you never hit these needs, you don't need Bazel, and you probably shouldn't use it because it is ferociously complicated. But if you do hit these needs, most other build tools simply do not cut it at all.
We're trying to support some of these use cases and provide a simpler alternative in https://mill-build.org, but Bazel really is a high bar to reach in terms of features that support large monorepos